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Executive Summary 



The NISE’s vision is that all students leave the 
educational system with an ability to make in- 
formed decisions about the matters related to 
science, mathematics, engineering, and technology 
(SMET) that they encounter in their daily lives. In 
this vision, all stakeholders in SMET education 
engage in active, contextual learning to acquire 
both a strong foundation in SMET and the ability 
to enhance that foundation as lifelong learners. The 
NISE, funded by the Education Human Resources 
Directorate (EHR) of the National Science Founda- 
tion, began work in July 1995. 

The NISE comprises students, staff, and faculty 
from the SMET and education colleges at the 
University of Wisconsin-Madison (UW) and the 
staff of the National Center for Improving Science 
Education (NCISE) in Washington, DC. The NISE 
also includes visiting Fellows representing the 
various stakeholder groups who are leaders in 
SMET education nationally and internationally. 

The UW and NCISE partnership includes a group 
of participants who, while educated in a wide 
variety of disciplines, share a common vision and 
goals for the NISE. It is exactly this multidisciplin- 
ary approach that has been absent from most 
education reform efforts to date. Having completed 
its first year of work, the NISE is off to a fast 
start. 

First year work was featured at NISE’s 1996 
Annual Forum in Washington, DC, developed by 
the Interacting with Professional Audiences team. 
Its cosponsors included National Science Teachers 
Association (NSTA), National Council of Teachers 
of Mathematics (NCTM), Delta Education, Dura- 
cell, Merck, and Toyota. The focus of the Forum 
was Professional Development. Billed as an oppor- 
tunity for diverse stakeholders to interact in signifi- 
cant ways, the Forum was well received by the 
community and was fully subscribed. Of the 
approximately 240 people attending, more than 
70% were from "outside the beltway" and repre- 
sented K-12, higher education, industry, and 
professional societies. 



The Communicating with Mass Audiences team 
has developed The Why Files, a Web site that 
explains the science behind the news. The Why 
Files uses recent headlines as a starting point for 
detailed descriptions of science, mathematics, engi- 
neering and technology issues. This effort allows 
us to explore and study the potential uses of cyber- 
space in enhancing both classroom teaching and 
SMET literacy for all. The site has received much 
attention. Since its inception in February 1996, the 
Why Files has earned virtually every major Web 
award and citation, including being named Micro- 
soft "site of the day" twice. The Why Files hit a 
major milestone the last week of July 1996 — 
downloading the millionth file. In addition, the 
Web site has been visited by people from 70 coun- 
tries. The Why Files were created so that NISE 
could study the use of the Internet to promote 
scientific learning. 

Part of the mission of the NISE is to provide a 
national leadership role in enhancing the quality of 
SMET education. This goal is being accomplished 
through a variety of mechanisms (e.g., scholarly 
publications, formal and informal presentations, 
materials on the Internet and the Web, and collabo- 
rative efforts). The NISE has collaborated with a 
number of other SMET organizations to provide 
leadership in SMET education. For example, we 
sponsored and arranged the kickoff meeting of 
Project EXTEND. The goal of this project, funded 
by the Exxon Education Foundation, is to extend 
the dialogue on the NCTM Standards. In addition, 
the NISE Evaluation and Policy Studies teams are 
collaborating with the Council of Chief State 
School Officers (CCSSO). The CCSSO is analyz- 
ing state policies and practices in mathematics and 
science and, in collaboration with the NISE, has 
held a conference on alignment and produced a 
paper on criteria forjudging alignment of Stan- 
dards/Frameworks and Assessment. We also are 
initiating collaborations with the informal science 
community through the Milwaukee Public Muse- 
um, a national leader in informal SMET education. 
We played a key role in the follow-up activities 



associated with the EHR February 1996 meeting on 
The Social Sciences Contribution to the EHR Un- 
dergraduate Review. 

There is a new and exciting opportunity to enhance 
the NISE’s leadership role nationally. In January 
1996, the Wisconsin Center for Education Re- 
search, home site for the NISE, was awarded the 
OERI National Research and Development Center 
on Achievement in School Mathematics and Sci- 
ence. The work of the OERI Center was designed 
to complement the NISE; it is directed by Thomas 
Romberg, a member of the NISE Management 
Team. The focus of the new center is on research 
and evaluation of classroom practice in K-12 
SMET education. We are excited about the possi- 
bilities for synergism between the new Center and 
the NISE. 



Organizational Structure 

The NISE is structured to be a highly interactive 
collegial system, while maintaining the clear lines 
of authority and responsibility necessary to ensure 
quality, accountability, direction, and leadership. 
The NISE actively fosters collaborative work and 
full cooperation among the education and science 
faculties at the University of Wisconsin-Madison 
and throughout the nation. The multidisciplinary 
nature of the NISE’s work is reflected in parity 
between scientists and educators throughout the 
management structure and in the composition of 
project teams. 

A 10-person Management Team consisting of a 
balance of scientists, education researchers, educa- 
tion practitioners, and representatives of the indus- 
trial sector meets quarterly to provide advice on the 
quality and direction of NISE work. 

An 18-person National Advisory Board consists of 
a rich and balanced mix of scientists, education 
researchers, education practitioners, and representa- 
tives from business, industry, government, and 
foundations. The Board provides advice and direc- 
tion to NISE’s work and enhances its visibility and 
impact. 



Fellows are recruited to the NISE and are an inte- 
gral part of the NISE project teams. Represented 
among the Fellows are all levels of colleagues, 
including K-12 teachers and professors and re- 
searchers from the postdoctoral level to the most 
senior ranks. This breadth and diversity enhances 
the quality of the discourse and maximizes the 
impact of the work of the Institute. There is no 
explicit requirement on length of stay at the NISE, 
but Fellows are generally in residence for periods 
of time sufficient to allow meaningful interaction 
with NISE faculty and staff. 

Research Programs 

The NISE is pursuing research in three areas of 
SMET education: Evaluation and Policy Studies, 
Professional Development, and College Level One. 

In the Evaluation and Policy Studies area are two 
teams. The goal of the Policy Analysis of System- 
ic Reform team is to understand how systemic 
reform in mathematics and science is being imple- 
mented and with what effects. In our work on 
systemic reform, we seek to know the nature of the 
evolving content and assessment standards; the 
politics surrounding development and implementa- 
tion of standards at the national, state, local, 
school, and classroom levels; the nature of the 
implementation process for restructuring within 
district and school; the degree to which classroom 
practice has changed toward that envisioned in the 
standards; and impacts on student learning, disag- 
gregated by basic and advanced skills as well as by 
race and income. We particularly want to imbed 
the emerging information on the NSF Systemic 
Initiatives within this broader context. The work is 
designed to address the information needs of EHR 
with its emphasis on systemic reform in mathemat- 
ics and science. 

The Strategies for Evaluating Systemic Reform 
team is well aware that systemic evaluation is an 
emerging field. Nobody knows how to evaluate 
systemic initiatives to determine their full impact 
or how information can best be used to advance 
such initiatives. The complexity of educational 
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systems, the difficulty in mounting coherent move- 
ment toward unifying goals, the number and range 
of influential forces, the fluctuation in the political 
and economic climates, the resources needed to 
sustain movement over time, and the lack of uni- 
formity of progress within a large system all con- 
tribute to the problem of determining the value and 
worth of these initiatives. Lack of knowledge about 
systemic evaluation is the central problem being 
addressed by the Strategies for Evaluating Systemic 
Reform project. The main goals of the project are 
to produce knowledge about systemic evaluation 
and determine how to do systemic evaluation. 

Professional Development is essential to effective 
educational reform. The EHR goal that every child 
in the United States has access to high-quality 
school education in science and mathematics can- 
not be realized without the availability of effective 
professional development for teachers. The intent 
of this project is to capture the learnings of current 
professional development efforts that will increase 
the knowledge base from both "craft wisdom" and 
disciplined inquiry and make that information 
accessible to practitioners and researchers alike. 

The project has three goals. The first is developing 
a framework for the design of professional learning 
opportunities for K-12 inservice science and mathe- 
matics teachers in order to expand the range of 
alternatives available to professional developers 
beyond the traditional format of workshops and 
institutes. The second goal of the project is to 
create a professional dialogue for elaborating and 
understanding the design framework and the issues 
raised in implementing, sustaining, and scaling up 
professional development learning opportunities. 
The third goal of the project is to create products 
that provide guidance to designers, funders, con- 
sumers, and evaluators of professional develop- 
ment. 

The College Level One (CL-1) team recognizes 
that high quality undergraduate programs in sci- 
ence, mathematics, engineering, and technology 
have a substantial impact on providing general 
SMET literacy for all students, preparing students 
for careers in SMET fields and in teaching, and in 
addressing equity issues. Experiences in first-year 
courses greatly influence career trajectories and 



lifelong attitudes toward SMET-related fields. The 
objective of the College Level One (CL-1) Team is 
to identify and study critical issues related to these 
courses. The team is addressing such issues as how 
first-year college courses in SMET can be made 
more attractive to students, add value for those 
who take them, and enhance the likelihood of 
success across the wide spectrum of students who 
enroll in them. 



Dissemination Programs 

The NISE is committed to disseminating its work 
to a wide range of audiences. 

The overarching goal of the Interacting with 
Professional Audiences team is to ensure that the 
new knowledge generated by the Institute gets into 
the hands of all professionals who can make use of 
it. The team helps Institute researchers identify and 
connect with relevant audiences beyond the aca- 
demic community and develop effective means of 
reaching them. As a result, a much broader range 
of professionals and their organizations will be 
knowledgeable enough to make use of Institute 
endeavors than is often the case for university- 
based research projects. The three primary goals of 
the Interacting with Professional Audiences team 
are (1) to share knowledge and information gener- 
ated by NISE with key SMET stakeholders, (2) to 
promote interaction and dialogue among SMET 
stakeholders about NISE research directions and 
applications, and (3) to encourage policymakers 
and practitioners to put research knowledge about 
SMET education into action. 

The Communicating with Mass Audiences team 
reached its first major milestone: developing and 
disseminating to the public a Web site that pro- 
motes SMET literacy by creating information pack- 
ages pegged to mass media headlines. The Why 
Files (the name is a play on the hit television pro- 
gram "The X-Files") combines skillfully written 
text, compelling graphics, timely news photos, and 
strategic linkages to other Web sites to produce a 
powerful communication tool to mass audiences. 

In Year 2, the team is developing a research effort. 
In Phase One, the research team is gathering de- 



scriptive data on Web communication patterns, 
using The Why Files audience members as pro- 
spective study subjects. In Phase Two, using the 
initial information gathered, the team will delve 
more deeply into questions about the effectiveness 
of communication via the Web. As a result of our 
inquiry, we hope to be able to suggest specific 
strategies for improving Web communication. 
These strategies will then be tested by our Why 
Files developers and studied again, using standard 
research and development processes. 

In addition, the Communicating with Mass Audi- 
ences team will increase awareness of the NISE by 
publicizing newsworthy deliverables, positioning 
NISE personnel as SMET education experts, and 
aggressively marketing The Why Files to increase 
its reach. 



Organizational Process Programs 

To enhance the overall efficiency and productivity 
of its work the NISE has two initiatives. 

The goal of the Cognitive Studies of Interdisci- 
plinary Communication team is to understand 
interdisciplinary collaboration and improve such 
collaboration within the NISE. We are reviewing 
research and conducting observational studies of 
NISE teams that will help us describe, understand, 
and ultimately design and implement new ways to 
facilitate the communicative processes that influ- 
ence productivity and quality of work in interdisci- 
plinary teams. We hypothesize that good collabora- 
tive teams will evolve toward a state of "collective 
intelligence," which entails functioning more like a 
coherent, intelligent organism than like a collection 
of disassociated, independent thinkers. We hyppth- 
esize that collective intelligence is a type of group 
behavior that might be promoted and enhanced 
through design and use of technologies that facili- 
tate idea processing during and between team 
meetings. 

The NISE Formative Evaluation team provides 
diverse forms of evaluation support for the Insti- 
tute. The Formative Evaluation team’s primary role 
is to provide internal "formative feedback" infor- 
mation to improve the Institute’s functioning as an 



agile and productive organization capable of achie- 
ving its stated and emerging goals. Secondary roles 
played by the Formative Evaluation team are to 
produce an evolving history of the organization; 
provide, gather, and manage information on organi- 
zational activity; and engage in collaborative work 
with various NISE teams. 



Summary 

In the first year of the NISE, we have: 

• initiated eight productive lines of research; 

• recruited a diverse group of Fellows to collabo- 
rate with us in pursuit of our mission; 

• made great strides in creating high quality 
interdisciplinary teams of researchers who work 
together effectively; 

• established increasingly productive collabora- 
tive relationships with EHR’s programs; 

• implemented and modified, where appropriate, 
our proposed management structure, including 
the creation of a highly visible National Advi- 
sory Board that will meet in Fall 1996; and 

• initiated collaborative work with a wide variety 
of professional organizations in SMET educa- 
tion. 

Our emphasis on dissemination is paying off with, 
for example, a fully subscribed first-year NISE 
Annual Forum and The Why Files on the Web 
being accessed by literally thousands daily. 

The EHR’s establishment of the NISE has created 
an enormous excitement across the SMET commu- 
nity. We have received hundreds of statements of 
interest, requests for information, and visitors to 
the Institute. Our mission is ambitious, but our 
commitment is great. At the end of five years, we 
expect to have launched a whole new approach for 
the continuous improvement of SMET education. 
The goal of high levels of SMET literacy for all 
segments of our population will have become 
better understood and more broadly accepted. New 
communities of scholarship and practice will have 
been established, where scientists, education re- 
searchers, and education practitioners work col- 
laboratively to attack the enduring problems of 
SMET education, problems that have resisted solu- 
tions from more narrow approaches. 
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Overview 



Institute Vision, Goals, and Strategies 



Institute Vision 

The National Institute for Science Education 
(NISE) is confronting head-on the challenges fac- 
ing our education system from kindergarten 
through graduate school. The NISE works with the 
Education and Human Resources Directorate 
(EHR) of the National Science Foundation (NSF) 
to transform the way science, mathematics, engi- 
neering, and technology (SMET) are taught. The 
complexity of this system with its multiple stake- 
holders requires our responsive, collaborative, and 
coordinated approach. 

The NISE’s vision is that all students leave the 
educational system with an ability to make in- 
formed decisions about the SMET-related matters 
that they encounter in their daily lives. In this 
vision, all stakeholders in SMET education engage 
in active, contextual learning to acquire both a 
strong foundation in SMET and the ability to en- 
hance that foundation as lifelong learners. There 
are three broad principles that form the basis for 
active, contextual learning and that guide our vi- 
sion for the NISE: 

1 . Science is a way of knowing and learning and 
not merely a body of already established facts. 
All students, not just academically elite or sci- 
ence track students, should understand how 
knowledge is generated as well as be knowl- 
edgeable about already established results. 

2. Students and teachers should be regarded as 
active participants in the construction of knowl- 
edge they "own," rather than as passive recipi- 
ents of knowledge that belongs to external au- 
thorities. 

3. The development of shared understanding is a 
complex, recursive process requiring continu- 
ous cooperation and collaboration among dif- 
ferent participants. 

While the American education system has deliv- 
ered an outstanding preparation for lifelong learn- 
ing to a select group of students — primarily to 
some students from the middle and upper class- 



es — most majority students and virtually all stu- 
dents from underrepresented groups have been 
inadequately prepared in the SMET areas. Our 
society cannot afford to waste this intellectual 
potential. We are working to ensure that all Ameri- 
cans receive the best possible education, especially 
in SMET. 

The NISE comprises students, staff, and faculty 
from the SMET and education colleges at the 
University of Wisconsin-Madison (UW) and the 
staff of the National Center for Improving Science 
Education (NCISE) in Washington, DC. The NISE 
also includes visiting Fellows representing the 
various stakeholder groups who are leaders in 
SMET education nationally and internationally. 

The UW and NCISE partnership includes a group 
of participants who, while educated in a wide 
variety of disciplines, share a common vision and 
goals for the NISE. It is exactly this multidisciplin- 
ary approach that has been absent from most edu- 
cation reform efforts to date. The overwhelming 
challenges facing SMET education cannot be effec- 
tively addressed by homogeneous teams consisting 
of either education researchers or SMET research- 
ers working independently. These communities 
have for too long operated on parallel paths with 
little interaction, to the detriment of SMET educa- 
tion. 

Institute Goals 

The NISE shares the larger goals of universal 
SMET literacy with the National Science Founda- 
tion (NSF) and the SMET community (NSF, 

1993). 

• Educate lifelong learners who are science and 
technology literate and workforce contributors. 

• Ensure access and opportunity for all from 
kindergarten to graduate school (K-GS). 

• Encourage a nationwide community of SMET 
and education researchers and practitioners to 
work collaboratively to continually strengthen 
SMET education. 
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• Work toward developing a seamless web of 
integrated experiences and expectations from 
kindergarten through college that prizes local 
adaptation, experimentation, and evaluation. 

The NISE’s primary work focuses on the more 
specific goals: 

• Enhance the knowledge base through improved 
indicators, student assessment, and program 
evaluation to support increased efficiency and 
effectiveness in learning. 

• Facilitate sharing of information about exem- 
plary programs and practices. 

• Conduct policy analysis at all levels of the 
education hierarchy to strengthen the policy 
environment for SMET education. 

• Work collaboratively with the NSF Education 
and Human Resources (EHR) Directorate to 
ensure maximum effectiveness of its programs 
and practices. 

• Provide demonstration sites for systemic reform 
in action. 

Institute Strategies 

We recognize that our broad vision cannot be fully 
achieved within a fixed time frame and a limited 
budget. However, we seek to make major advances 
by using the following strategies: 

1 . Operating through a partnership involving a 
university that provides nationally recognized 
researchers and reformers from its SMET col- 
leges and its School of Education; a Washing- 
ton, DC, site that facilitates communication 
with NSF and allows us to host conferences 
that are easily accessible to key federal policy- 
makers; and the NSF’s EHR Directorate. 

2. Fielding a team of Pis, having parity between 
SMET researchers and education researchers, to 
ensure that the maximum experience, under- 
standing, and wisdom are brought to bear on 
complex problems. 

3. Actively involving K-12 teacher collaborators 
in our work. 



4. Bringing visiting Fellows to both sites who 
represent the best in the SMET education field 
nationally and internationally. 

5. Collaborating with business and industry. As 
employers of most of our graduates, exemplars 
of work-place education, and providers of 
student practicum sites, they are key stake- 
holders in SMET education. 

6. Focusing on key points of leverage in the 
K-GS system. 

7. Providing national leadership by forming alli- 
ances with key organizations and individuals 
across the country, including scientific, engi- 
neering, and professional societies, education 
research societies, K- 1 2 teacher organizations, 
leaders of equity action groups, accreditation 
agencies, and local, state, and federal govern- 
ment officials and agencies. 

8. Utilizing demonstration sites for testing and 
evaluating NISE processes and products. 

9. Forming network alliances. 

We are forming network alliances to work on 
selected problems facing SMET education reform. 
Since reform is so complex, it requires active par- 
ticipation of representatives of all stakeholders in 
NISE projects. To achieve successful educational 
reform, communication among NISE staff and 
stakeholders must shift from unidirectional knowl- 
edge transmission (where experts tell teachers how 
to do it) to multidirectional interactions involving 
knowledge sharing and negotiation (where network 
alliances will involve multidisciplinary collabora- 
tions among members of the NISE, selected NISE 
partner project leaders, and relevant stakeholders- 
— all of whom focus on a particular problem). 
Network alliance members communicate with each 
other through site visits, intensive conferences, and 
technological media (e.g., video- or teleconferenc- 
ing, the Internet). The intended outcomes of the 
network alliances are the syntheses of core princi- 
ples that underlie successful reform in the NISE’s 
focus areas and, ultimately, the desired cultural 
change in SMET education. 
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Highlights of NISE Work to Date 



Having completed its first year of work in July of 
1996, the NISE is off to a fast start. Details of 
accomplishments and plans are described below, 
but some highlights are mentioned here. For exam- 
ple, four teams have held national workshops to 
ensure that their efforts are informed by the larger 
community (College Level One, National Policy 
Studies of Systemic Reform, Professional Develop- 
ment, and Strategies for Evaluating Systemic Re- 
form). A number of the teams have prepared schol- 
arly synthesis articles on key topics, with some of 
these papers authored by NISE Fellows. 

The Year 1 work was featured at the 1996 Annual 
Forum in Washington, DC, sponsored by the NISE. 
Its cosponsors included National Science Teachers 
Association (NSTA), National Council of Teachers 
of Mathematics (NCTM), Duracell, Merck, and 
Toyota. The focus of the Forum was Professional 
Development. Billed as an opportunity for diverse 
stakeholders to interact in significant ways, the 
Forum was well received by the community and 
was fully subscribed. Of the approximately 240 
people attending, more than 70% were from "out- 
side the beltway" and represented K-12, higher 
education, industry, and professional societies. 
Active recruitment and inclusion of diverse constit- 
uencies exemplifies the commitment to community 
building that is being pursued by all NISE teams. 

The Communicating with Mass Audiences team 
has developed The Why Files, a Web site that 
explains the science behind the news. The Why 
Files uses recent headlines as a starting point for 
detailed descriptions of science, mathematics, engi- 
neering and technology issues. This effort allows 
us to explore and study the potential uses of cyber- 
space in enhancing both classroom teaching and 
SMET literacy for all. The site has received much 
attention. Since its inception in February 1996, the 
Why Files has earned virtually every major Web 
award and citation, including being named Micro- 
soft "site of the day" twice, most recently by Mi- 
crosoft United Kingdom in July 1996; Microsoft 
Network’s "Pick of the Day," Netscape’s "What’s 
Cool" as recently as July 19, 1996, and rated a 
Four Star sight — the highest rating — by Magellan, 
and Magellan and Yahoo’s Picks of the Week. The 



Why Files received the Eisenhower National Clear- 
inghouse Digital Dozen, the Too Cool Award, and 
the Blue Planet Award. The Why Files has also 
been cited in New Scientist: Planet Science, and 
reviewed by Hot Wired and CNN Computer Con- 
nection (June 1, 1996). The Why Files hit a major 
milestone the last week of July 1996 — we down- 
loaded our millionth file. In addition, the Web site 
has been visited by people from 70 countries, and 
as many as 100,000 files per day are being down- 
loaded by users. The Why Files were created so 
that NISE could study the use of the Internet to 
promote scientific learning. 

Part of the mission of the NISE is to provide a 
national leadership role in enhancing the quality of 
SMET education. This goal is being accomplished 
through a variety of mechanisms (e.g., scholarly 
publications, formal and informal presentations, 
materials on the Internet and the Web, and collabo- 
rative efforts). The NISE has collaborated with a 
number of other SMET organizations to provide 
leadership in SMET education. For example, we 
sponsored and arranged the kickoff meeting of 
Project EXTEND. The goal of this project, funded 
by the Exxon Education Foundation, is to extend 
the dialogue on the NCTM Standards. In addition, 
the NISE Evaluation and Policy Studies teams are 
collaborating with the Council of Chief State 
School Officers (CCSSO). The CCSSO is analyz- 
ing state policies and practices in mathematics and 
science and, in collaboration with the NISE, has 
held a conference on alignment and produced a 
paper on criteria for judging alignment of Stan- 
dards/Frameworks and Assessment. We also are 
initiating collaborations with the informal science 
community through the Milwaukee Public Muse- 
um, a national leader in informal SMET education. 
We played a key role in the follow-up activities 
associated with the EHR February 1996 meeting on 
The Social Sciences Contribution to the EHR Un- 
dergraduate Review. The NISE also coordinates the 
EHR Special Emphasis Panel on Evaluation as 
described in the Evaluation and Policy Studies 
section below. 

There is a new and exciting opportunity to enhance 
the NISE’s leadership role nationally. In January 
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1996, the Wisconsin Center for Education Research 
(WCER), home site for the NISE, was awarded the 
National Research and Development Center on 
Achievement in School Mathematics and Science 
by the U.S. Department of Education’s Office of 
Educational Research and Improvement (OERI). 
The work of the OERI Center was designed to 



complement the NISE; it is directed by Thomas 
Romberg, a member of the NISE Management 
Team. The focus of the new center is on research 
and evaluation of classroom practice in K-12 
SMET education. We are excited about the possi- 
bilities for synergism between the new Center and 
the NISE. 



Organization and Management Structure 



The NISE is structured to be a highly interactive 
collegial system, while maintaining the clear lines 
of authority and responsibility necessary to ensure 
quality, accountability, direction, and leadership 
(see Figure 1). The NISE actively fosters collabo- 
rative work and full cooperation among the educa- 
tion and science faculties at the University of Wis- 
consin-Madison and throughout the nation. Ac- 
cordingly, one director of the NISE is a SMET 
researcher and the other an education researcher. 
The multidisciplinary nature of the NISE’s work is 
reflected in parity between scientists and educators 
throughout the management structure and in the 
composition of project teams. 

Building organizational procedures is essential for 
any new national research institute. In the case of 
the NISE, however, the development of a function- 
ing institute is something worth documenting and 
studying as well. NISE’s multidisciplinary ap- 
proach has long been called for in SMET education 
but largely absent from education reform efforts to 
date. Much progress has been made in establishing 
truly effective multidisciplinary work. A multidisci- 
plinary approach characterizes not only the re- 
search teams but the entire organizational structure 
and decision-making process of the NISE. The 
development of the NISE as a multidisciplinary 
organization is being carefully documented through 
the Formative Evaluation efforts led by Susan 
Millar and the Cognitive Studies of Interdisciplin- 
ary Communication led by Sharon Derry (both are 
described later). We believe that their findings will 
facilitate the organization and functioning of other 
such centers. 

As originally proposed, a ten-person Management 
Team consisting of a balance of scientists, educa- 
tion researchers, education practitioners, and repre- 



sentatives of the industrial sector has been created. 
Led by the co-directors, the Management Team 
meets quarterly to provide advice on the quality 
and direction of NISE work. 

The Team Leaders Team is a mechanism bringing 
the co-directors, the project manager, and the team 
leaders together on a regular basis to share prog- 
ress, strengthen communications, and make deci- 
sions about the implementation of NISE strategies. 
The Team Leaders Team has established itself as 
the chief mechanism for coordinating and integrat- 
ing the NISE scope of work. 

The NISE has established an 18-person National 
Advisory Board co-chaired by Wisconsin Governor 
Tommy Thompson (former head of the National 
Governors’ Association and the Education Com- 
mission of the States) and John Porter, CEO of the 
Urban Education Alliance and former Michigan 
Superintendent of Public Instruction. Like all parts 
of the NISE, the National Advisory Board consists 
of a rich and balanced mix of scientists, education 
researchers, education practitioners, and representa- 
tives from business, industry, government, and 
foundations. The Board is to provide advice and 
direction to NISE’s work and to enhance its visibil- 
ity and impact. 

Fellows are recruited to the NISE and are an inte- 
gral part of the NISE project teams. A call for 
applications and nominations for NISE Fellows has 
been widely circulated through the NISE Web site 
and through direct mailings. Announcements of the 
application process have also appeared in key 
professional journals and magazines. 

NISE is recruiting a diverse group of Fellows from 
a wide variety of stakeholder groups, including 
K-12 teachers, higher education faculty in SMET 
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and education research, industry, private founda- 
tions, the media, and government. Represented 
among the Fellows are all levels of colleagues, 
including K-12 teachers and professors and re- 
searchers from the postdoctoral level to the most 
senior ranks. This breadth and diversity enhances 



the quality of the discourse and maximize the 
impact of the work of the Institute. There is no 
explicit requirement on length of stay at the NISE, 
but Fellows are generally in residence for periods 
of time sufficient to allow meaningful interaction 
with NISE faculty and staff. 



Research Programs 

Evaluation and Policy Studies 



Policy Analysis of Systemic Reform 



The goal of the Policy Analysis team is to generate 
useful knowledge about implementing systemic or 
standards-based reform in mathematics and science 
education. The present moment is critical in the 
national effort to become "first in the world in 
mathematics and science." The standards them- 
selves have taken the vital first step by providing a 
detailed vision of a vastly improved system, com- 
plete with benchmarks of achievement at each age 
level, and have attracted a high degree of consen- 
sus and support among a broad group of scientists 
and educators. 

But the next stage of implementation is very chal- 
lenging. While the standards provide guidelines, 
they do not explain the sequence of courses and 
instruction that will satisfy these standards. State 
and local policymakers and professional groups 
must "reinvent” the standards in actual practice, in 
most cases without the same rich level of expertise 
that was available in developing the standards. 

Year 1 Accomplishments 

With limited resources and an enormous amount to 
leam, the challenge for the Policy Analysis team is 
to carve out an efficient research agenda. Year 1 of 
the project was spent casting a broad net, apprais- 
ing the state of our knowledge about reform, and 
understanding the obstacles. Michael Knapp wrote 
a paper on the implementation of systemic reform; 
Michael Kirst on the politics of standards; and 
several authors (policy analysts, mathematicians 
and scientists) wrote papers on the national stan- 
dards themselves. William Clune attended meetings 
of the systemic initiatives in Washington, DC; and 
Allan Odden, the original systemic reform leader, 
was appointed by Governor Tommy Thompson to 
a newly created Wisconsin Education Task Force 
on standards-based reform in Wisconsin. Governor 
Thompson is immediate past chair of both the 
National Governors’ Association and the Education 
Commission of the States and led a Summit of 



governors and business leaders on standards-based 
reform in March 1996. 

The research conducted in Year 1 culminated in a 
"summer seminar" held July 24-25 in Madison, 
attended by a broad group of reformers, policy 
analysts, evaluators, and experts in the standards 
and their subject matters. William Clune wrote a 
paper synthesizing the problems identified as cen- 
tral in the seminar and suggesting how the future 
research agenda could be shaped to deliver the 
most useful knowledge about reform. Clune’ s 
paper deals with two general issues summarized 
below: (1) how to think about systemic reform as 
an effort to produce substantial gains in student 
achievement; (2) how to design the research in 
Year 2 and beyond to produce knowledge that is 
useful in improving systemic reform in mathemat- 
ics and science education. 

What Is Systemic Reform? 

Systemic reform is the creation of deep and broad 
gains in student achievement through a substantial 
upgrading of instruction (systemic change), pro- 
duced through the cumulative and coherent influ- 
ence of multiple policy instruments (systemic poli- 
cy). The "deep and broad" aspect of systemic re- 
form can be contrasted with prior reforms that have 
been either broad and shallow (for example, a high 
school graduation requirement of three rather than 
two mathematics courses) as well as "deep and 
narrow," (for example, tremendous improvement in 
one or a few schools or among a few teachers). 

How Systemic Is Enough (How Deep and How 
Broad) ? 

If systemic reform is characterized by depth 
and breadth of change, a critical question is, How 
deep and how broad? A working definition of 
systemic change for any one change effort would 
be that the achievement of at least 10 percent of 
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the students in an area improves by at least one 
performance benchmark on an appropriate test. 
Systemic reform can then be defined as any set of 
interventions achieving this result. The 10 percent 
threshold is suggested as a way to give some teeth 
to the definition of reform, without requiring im- 
mediate universal change, and is guided by the 
example of the Advanced Placement exams that 
function to raise the achievement of about 10 per- 
cent of the nation’s students about one full grade 
level (that is, from 1 2th grade to first-year college). 
Despite some disagreements about content, most 
people would consider the AP an effective system. 

What Kind of Multiple Policy Instruments? 

There has been a broadening in our understand- 
ing of what combinations or packages of policy 
instruments are capable of producing broad and 
deep systemic change. The "traditional model" of 
policy alignment is still viable (alignment among 
curriculum guidelines, student assessments, and 
professional development) and is being implement- 
ed in various places, for example, Kentucky, Char- 
lotte-Mecklenberg, and Dallas. A second model in 
common practice is the "network" of professional 
development and school restructuring. School im- 
provement networks such as "Success for All" have 
changed instruction and achievement in many 
schools, and networks of professional development 
in California predated policy alignment and sur- 
vived even when the state testing system collapsed. 
Less is know about the potential of national (but 
not governmental) systems of curriculum and test- 
ing (such as Advanced Placement) and powerful 
(widely accepted) curriculum models and text- 
books. Obviously it is possible for these systems to 
reinforce each other, and many "systemic initia- 
tives" do consist of such combinations. 

Year 2 Activities 

Given this concept of systemic reform, what re- 
search projects can deliver the most powerful 
knowledge about how to bridge the gap between 
standards and practice? Below is the research agen- 
da that emerged from the review of systemic re- 
form conducted by the Policy Analysis team in 
Year 1. 



The Year 2 research agenda has already been ap- 
proved by NSF, and the following four research 
projects will be completed during this year. 

1. Centralized and decentralized systemic policies 

Research question: Are there decentralized and 
nongovernmental alternatives to the "classic" vision 
of systemic policy that are equally capable of pro- 
ducing systemic change? 

Policy Problem: The classic version of systemic 
policy (alignment of official curriculum guidelines, 
student assessments, professional development, and 
texts) does not appear to be the only kind of sys- 
tem that produces systemic change (large scale 
upgrading of instruction and student achievement). 
Policymakers should be aware of these alternatives, 
especially because the classic model is not always 
available or effective, and the alternatives may be 
less expensive. 

Methodology: Commission a major paper on a 
variety of systems which have demonstrated a 
capacity for systemic change. Analyze the common 
features of more or less "complete" systemic poli- 
cies that have demonstrated some success in vari- 
ous parts of the country, regardless of whether the 
policies are part of a systemic initiative or are 
oriented toward the new standards (thus distin-- 
guishing the efficacy of instructional guidance 
from the particular content being advanced). Com- 
mon features analyzed will include source of 
"alignment at the bottom" (method of acquiring 
influence over correspondence of curriculum, tests, 
texts, and training), plus incentives to participate. 
Examples could include: (a) the Advanced Place- 
ment system, (b) state- and districtwide reform 
efforts (such as Kentucky and Charlotte-Mecklen- 
berg), (c) successful school improvement and pro- 
fessional development networks that show some 
evidence of going to scale (e.g., Success for All, 
Annenberg, New American Schools), and (d) infor- 
mal links between textbook publishers, teachers, 
and schools. 

This analysis will look not only for the key policy 
ingredients (such as coherence, organizational 
capacity, and incentives and evaluation systems) 
but also what Allan Odden called "management" 
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— what makes the policies work together toward 
coherent outcomes. Attention also will be paid to 
verification of systemic effectiveness (what is the 
evidence that systemic change has occurred). 

2. A "systemic change power rating " of systemic 
reforms in the systemic initiatives 

Research question: Based on knowledge acquired 
to date, what combinations of policy instruments 
adopted by the systemic initiatives appear most 
likely to produce systemic change? 

Policy problem: The systemic initiatives are de- 
signed to produce change and knowledge about the 
effectiveness of systemic policy, and some initia- 
tives have been judged more effective than others. 
But knowledge about effective/ineffective policies 
has not been systematized and widely shared in a 
way that can guide policymakers. 

Methodology: Commission a major paper that will 
develop and validate a system for rating the "pow- 
er" of the NSF systemic initiatives, including the 
probable impact on teaching and learning, tradeoffs 
between long and short run change, and breadth vs. 
depth. This project will refer to the criteria already 
utilized by NSF in evaluating the promise and 
success of systemic reform and also on the knowl- 
edge and judgment of principal investigators and 
program officers, as well as the overlapping work 
of the Evaluation unit within NISE. 

3. Adjustments in the systemic initiatives due to 
local political context 

Research question: What kinds of adjustments in 
the policy interventions of systemic initiatives have 
been necessary because of different political con- 
texts and cultures? 

Policy problem: It is apparent from the summer 
seminar and other discussions that most, if not all, 
systemic initiatives put a major effort into adjust- 
ing the structure of their policy interventions to the 
local context. For example, Uri Treisman, the 
Principal Investigator of the Texas Statewide Sys- 
temic Initiative, explained the impact of state de- 
centralization on his project. The problem is that 
the classical model of systemic policy did not 



discuss the details of such adaptation, and there has 
been little systematic knowledge produced about it 
(although studies of systemic reform do reach some 
generalizations). 

Methodology: Knowledge about the dynamics of 
systemic initiatives is possessed mainly by those 
involved with them, and local circumstances are 
often fluid and dynamic. Consequently, this project 
can best be handled by a new problem-solving 
group consisting of staff from NISE, NSF, and the 
systemic initiatives. Small papers on political strat- 
egies and policy adjustments will be issued by the 
group on the basis of discussions held bimonthly in 
Washington, DC. 

4. Equity interventions that produce systemic 
change in course work and student achievement 

Research question: What educational programs can 
be identified that, by reputation, have substantially 
upgraded instruction in mathematics and science 
and raised student achievement, for a substantial 
number of poor, minority, and female students? 

Policy problem: A substantial amount of data col- 
lection and analysis has demonstrated persistent 
gaps between the achievement of groups in our 
society, and many programs have been addressed 
to closing these gaps, but there seems to be little 
knowledge about the relative success, program 
content, and generalizability of these programs. 
Ultimately, knowledge about how to close the gap 
is more useful than knowledge about the gap itself. 

Methodology: Work in Year 2 will be concerned 
with identifying promising programs, while empiri- 
cal investigation of the programs can occur in later 
years. A new working group on equity in student 
achievement will bring together staff from four 
centers housed in WCER that are working on simi- 
lar issues. This group will help identify programs 
with demonstrated success in producing systemic 
change in mathematics and science education and 
share knowledge about the equity issue garnered 
from different contexts. Perhaps selected staff from 
Statewide and Urban Systemic Initiatives will be 
added to this group on an occasional basis for 
designated topics. 
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Beyond Year 2 Projects 

1. Professional and social consensus/dissensus 
around the national standards in mathematics and 
science 

Research question: What are the main issues divid- 
ing professionals and others about the national 
standards in mathematics and science, and how 
much is genuine disagreement as opposed to mis- 
understanding or exaggeration? 

Policy problem: Professional and social disagree- 
ments about the standards have begun to hinder the 
process of reform (for example, the termination of 
state student assessments in California). However, 
despite widely publicized disagreements, the sum- 
mer seminar suggested that there is a high degree 
of consensus among both educators and science 
professionals that the mathematics and science 
standards are correct in continuing much that has 
always been taught, such as exact calculation and 
familiar subject matters and topics, as well as an 
increased emphasis on understanding, scientific 
thinking, and problem solving. Both traditionalists 
and innovators are interested in some increased 
degree of "constructivism" in student learning, at . 
least to the extent that students should understand 
what calculations and procedures "mean" as well as 
how to execute them. The relative emphasis on 
applications in different positions also seems to 
have been exaggerated. On the other hand, areas of 
disagreement that do exist are interesting and im- 
portant and might be useful to policymakers and 
the public in making decisions. Unfortunately, the 
national debate has become both ideologically 
polarized and factually confusing. 

Methodology: A scholarly and cogently written 
paper could explore areas of agreement, dispel 
mistaken impressions and beliefs, and explain how 
true and false impressions have become exaggerat- 
ed and politicized, as well as being honest about 
the existence of continuing disputes and the man- 
ner in which policy has sometimes gone astray. A 
possible method of comparing different positions 
would be to assess the similarity of new curricula 
meeting the standards with different aspects of 
current curricula and courses. 



2. The overlap between new standards and old 
assessments 

Research question: What can policymakers learn 
from existing assessments about the progress of 
standards-based reform? 

P olicy problem: Standardized student assessments, 
such as NAEP (the National Assessment of Educa- 
tional Progress), SATs (Scholastic Assessment 
Tests), the AP (Advanced Placement) tests, are of 
potentially great significance in the systemic re- 
form movement for three reasons: (1) on content, 
they translate general content standards into specif- 
ic learning outcomes; (2) on performance, they 
spell out the age at which students are expected to 
acquire the specified knowledge and skill; (3) on 
progress, they provide a readily available measure 
of any gains in student learning. This commend- 
able specificity of assessments is exactly why they 
become sources of disagreement. One disagreement 
that could benefit from immediate clarification is 
how much existing measures of student achieve- 
ment, which do have age-graded benchmarks, over- 
lap with the new national standards. For example, 
the NAEP, AP, and international tests all make 
comparisons of groups against performance bench- 
marks. If a systemic initiative moved 10 percent of 
students up one benchmark on the NAEP, would 
that be the kind of progress aimed for by the stan- 
dards? We know that the standards aim for some 
qualities not measured by these tests, but is there 
also an area of overlap? For policymakers who rely 
on these tests, the existing polarized debate of 
"good test" and "bad test" must be replaced with 
"how good in what way." 

Methodology: A synthesis paper could be written 
with two objectives: (a) explain the degree to 
which existing and widely used measures of stu- 
dent achievement are and are not acceptable indica- 
tors of progress in systemic reform (e.g., NAEP, 
international comparisons, SATs, Iowa and Stan- 
ford achievement tests); and (b) explain what addi- 
tional dimensions will be measured by new assess- 
ment systems under development (for example, by 
the New Standards project), when these might 
become available, and the implications for teaching 
and learning. 
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3. A comparison of ideal standards-based curricu- 
la with existing curricula 

Research question : How much would the existing 
mathematics and science curricula of typical 
schools need to change to match curricula that 
would be considered ideal (or completely satisfac- 
tory) under the standards? 

Policy problem: Systemic initiatives and other 
reform efforts often find themselves in the situation 
of implementing the standards without specific 
guidelines. The result often is superficial change or 
a repackaging of the existing curricula. Many com- 
mentators have noted how useful it would be to 
have a more specific idea of the goals of reform, 
and a comparison of curricula is one way of illus- 
trating this contrast. 

Methodology: The focus of this project should not 
be on the standards themselves but on a compari- 
son of the standards with existing practice includ- 
ing curriculum, teaching, and assessments. The 
focus cannot be limited to curriculum in the sense 
of scope and sequence, because we would also 
need to understand the demands that new material 
would place on the classroom practice of the exist- 
ing teaching force. But, to keep the project from 
becoming too ambitious at first, a useful starting 
place might be to assume implementation of some 
available acceptable curriculum (perhaps in the 
form of model textbooks), make some assumptions 
about the learning goals for students at different 
ages, and ask what extra training, if any, teachers 
in typical school systems would need to teach in 
such a curriculum (for example, the possible prob- 
lem of elementary school teachers teaching higher 
levels of mathematics at earlier grades). 

4 . A " Good Housekeeping " system for rating 
mathematics and science textbooks 

Research question : How well do mathematics and 
science textbooks currently in use measure up to 
the new national standards? 

Policy problem: Textbooks are influential in deter- 
mining course content, and many existing texts are 
claiming some degree of conformity to the new 
national standards. But on close inspection many of 



0 




the changes made in the textbooks supposedly to 
meet the standards are superficial and would re- 
quire little if any change in practice. A rating of 
texts by an independent group against the standards 
thus could be a big help in the central task of be- 
coming more specific about reform. There is no 
system presently in existence for rating textbooks 
against national standards, although several states 
do go through an exercise of approving texts 
against state standards. Since several national orga- 
nizations have or plan projects for comparing text- 
books with the standards, NISE can build on and 
coordinate with these efforts in analyzing the use- 
fulness of a full blown rating system. 

Methodology: A central, authoritative system for 
rating textbooks would represent a big change in 
American education. The first step could be a 
paper that assesses the advantages and disadvan- 
tages of such a system, making reference to the 
textbook studies of national organizations, state ap- 
proval systems, and the efforts of publishers them- 
selves to meet new standards as they emerge. 

5. New ground rules for future systemic initiatives 

Research question: Should future versions of cen- 
tralized policies be designed to encourage systemic 
reform be modified in light of what we have 
learned about existing efforts? 

Policy problem: The current wave of NSF systemic 
initiatives was launched in a framework that en- 
couraged a wide variation in approach by states 
and localities (e.g., desire to encourage experiments 
and avoid federal control). But the SI program is 
now well under way, criteria of effectiveness have 
gradually been developed, data reporting require- 
ments have been changed, and some programs 
have been judged more meritorious than others. 

This raises the question of whether future programs 
should set more stringent conditions of participa- 
tion, for example, with a better focus on making 
real changes in student achievement. There are also 
parallel efforts to stimulate broad scale reform of 
mathematics and science education, not part of any 
NSF funded systemic initiative, such as the Gover- 
nors’ summit and the Annenberg projects. Policy- 
makers at all levels could benefit greatly from 
understanding the accumulated wisdom of these 
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various efforts about the conditions of greater and 
lesser success. 

Methodology: A paper analyzing the structure of 
successful reform initiatives could be helpful not 
only in identifying the components of success but 
also practical problems such as incentives for 
states, localities, and schools to participate under 
stringent conditions and the political problems that 
might be involved with a stronger central role and 
potentially fewer grant recipients (at least at first). 
The paper should not only identify problems but 
also consider alternative ways of solving them. A 
broad range of reform initiatives should be exam- 



ined (NSF’s Systemic Initiatives, independent state 
and local initiatives, Annenberg, etc.). 

Staff 

The core includes William Clune, team leader; 
Allan Odden; Andrew Porter; William Tate; Al- 
berto Rodriguez; Ron Jetty; Paula White; Bassam 
Shakhashiri, and Janice Downer (also on the Col- 
lege Level One NISE team); John Wright, Judy 
Roitman, Richard Rossmiller, Deborah Tepper 
Haimo, Senta Raizen, Michael Kirst, and Michael 
Knapp. We will attempt to schedule meetings to 
allow Uri Treisman to meet with our team when he 
visits the campus quarterly. 



Strategies for Evaluating Systemic Reform 



Systemic evaluation is an emerging field (Chubin, 
1995). Nobody knows how to evaluate systemic 
initiatives to determine their full impact or how 
best information can be used to advance such 
initiatives. The complexity of educational systems, 
the difficulty in mounting coherent movement 
toward unifying goals, the number and range of 
influential forces, the fluctuation in the political 
and economic climates, the resources needed to 
sustain movement over time, and the lack of uni- 
formity of progress within a large system all con- 
tribute to the problem of determining the value and 
worth of these initiatives. Lack of knowledge about 
systemic evaluation is the central problem being 
addressed by the Strategies for Evaluating Systemic 
Reform project. The main goals of the project are 
to produce knowledge about systemic evaluation 
and determine how to do systemic evaluation. 

Over the past five years, some progress has been 
made in learning about systemic reform by tho§e 
engaged in evaluating the state systemic initiatives 
(Shields, Corcoran, & Zucker, 1994; Zucker & 
Shields, 1995; Zucker, Shields, Adelman, & Po- 
well, 1995). Reports of these studies are primarily 
descriptive information produced using traditional 
techniques. Most of these techniques and methods 
come from program evaluation (Scriven, 1993). 
Little attention has been given to judging how the 
system has "gone to scale," affecting practices in a 
significantly large number of classes and resulting 



in systemic changes in students’ learning of sci- 
ence, mathematics, and technology. Absent were 
methods for detecting large system changes and 
attributing these changes to the reform initiatives. 

This evaluation project is producing thoughtful 
analyses and studies needed to advance the field of 
systemic evaluation. The work of the project ad- 
dresses directly the new challenges that evaluating 
large educational systems present: large numbers, 
time frame, colinearity, attribution, achievement 
gaps, and metrics for measuring systemic progress. 

Year 1 Accomplishments 

Information about the 24 Statewide Systemic Ini- 
tiatives (SSIs) was collected and aggregated into 
two reports. 

1. An abbreviated case study of one state’s system- 
ic reform activities was written. 

2. The evaluators of each existing SSI were asked 
to send us their most recent evaluation plan. Re- 
sponses were received from 16 of the 24 states. 

The information supplied by the evaluators was 
condensed into a matrix. Each row represents a 
state. The columns represents the goals of the SSI 
and the evaluation strategies and techniques used to 
determine whether the goals are being met. A 
paper "Purposes and Issues of Systemic Evaluation 





in Education" was prepared, drawing on the infor- 
mation received on the evaluations of the SSIs and 
existing literature. Four purposes for systemic 
evaluation were apparent from this analysis: design 
evaluation, management evaluation, leverage evalu- 
ation, and verification evaluation. The identified 
critical issues for systemic evaluation were system- 
wide impact, time frame, instrumentation, equity, 
and criteria for judging merit. 

An expanding NISE library has been created of 
documents and reports related to the SSIs and their 
evaluations. In addition to the collection of reports, 
a bibliography has been created with over 125 
entries of references relevant to systemic reform 
and its evaluation. Many of the references are in 
the library. This bibliography will be made avail- 
able on the Web. An index will be created to make 
the library more accessible and useful to those 
engaged in doing evaluations. 

A successful two-day conference was conducted on 
January 4-5, 1996. Twenty-six people from across 
the nation representing different roles related to 
SMET met to identify questions for and about 
evaluation of systemic reform. Questions for an 
evaluation are those that would be answered by an 
evaluation. The questions about an evaluation are 
those that need to be considered in designing an 
evaluation. Four discussion papers were prepared 
for the conference. Each paper enumerated a list of 
questions for and about evaluation centering on a 
particular area of systemic reform. The conference 
proceedings are now available. 

The arduous task of forming a multidisciplinary 
network of people interested in the issues related to 
evaluation of systemic reform was initiated. Those 
who attended the conference along with another 20 
people who could not attend have expressed inter- 
est in the work being done. This group of nearly 
50 constitutes a beginning network that will be 
expanded. One function of this group will be to 
foster greater dissemination of products produced 
by NISE in general and the Strategies for Evaluat- 
ing Systemic Reform project in particular. 



Year 2 Activities 

Two main objectives have been set for Year 2: (1) 
Continue development of strategies for evaluating 
systemic education reform: (2) Expand the scholar- 
ly attention given by leading evaluation experts 
from a number of disciplines to help resolve criti- 
cal problems related to the evaluation of systemic 
education reform. The project is building on ideas 
and information produced in Year 1 — namely, 
questions generated for and about evaluations of 
systemic reform; alignment of standards, frame- 
works, and assessments; and the state of existing 
evaluation practices of systemic initiatives. 

Conceptualizing Approaches for Evaluating Sys- 
temic Reform 

Four major areas rfelated to developing models for 
evaluating systemic reform are being addressed. 

Lessons Learned from Statewide Systemic Initiative 
(SSI) Evaluations. Dr. Charles Bruckerhoff, an 
evaluator of Connecticut’s SSI, is producing a 
report abstracting lessons learned from the existing 
evaluations of the SSIs. What is learned will be 
applied to the evaluation of the Urban Systemic 
Initiatives (USIs) and the Rural Systemic Initiatives 
(RSIs). Dr. Bruckerhoff is serving as an NISE 
Fellow. He is taking into consideration the differ- 
ent contexts for doing evaluations within the SSIs 
and across the SSIs. The conditions, strategies, and 
instruments for doing evaluation at the state level 
are being related to the context, goals, and resourc- 
es for the USIs and the RSIs. 

Methods for Describing Change in Educational 
Systems. Jim Ridgway, Professor at Lancaster 
University in the United Kingdom, is writing a 
monograph building on the assumption that more 
than one view of change is appropriate in judging 
the value of systemic initiatives. Research methods 
and tools from a range of fields relevant to educa- 
tion are being identified. For example, how re- 
searchers in epidemiology determine causal rela- 
tions within large arid dynamic systems is being 
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related to causal analyses of educational systemic 
initiatives. National mathematics and science cur- 
riculum standards are being used as a frame of 
reference for judging and exploring the application 
of methodologies from other fields of inquiry to 
evaluating systemic reform. 

Overcoming the Time- Frame Issues in Detecting 
Systemic Outcomes. Positive evidence of a system- 
ic initiative may not be detectable simply because 
not enough time has elapsed for teachers to synthe- 
size the change and make adjustments in their 
classroom practices, for students’ activities to have 
changed, and for student outcomes to be different. 
Time-frame issues are confounded by the complex- 
ity of the instructional process in SMET, the size 
of the system, and the nature of organized educa- 
tion. Effects of systemic reform over time may not 
grow linearly, but exponentially or discretely. For 
example, significant effects that grow exponentially 
may be less detectable in the early years, but have 
the potential of eventually expanding to a very 
large number. Or, effects within a few isolated 
districts may expand over time to other districts. 
Systemic evaluations need to be sensitive to differ- 
ent growth models of systemic educational change. 
Projections, modeling, and other techniques will be 
explored as possible techniques for attending to 
time-frame issues. Viable analytic methods will be 
explored from other fields that have dealt with 
time-frame issues. For example, epidemiologists 
use the term "analytic horizon" to identify the time 
period to be considered with regard to costs and 
benefits of health outcomes that occur as a result 
of an intervention (Center for Disease Control and 
Prevention, 1995). An analogous time period may 
be useful in considering the impact of educational 
initiatives. 

Equity and Gap Metrics. Jane Butler Kahle, Condit 
Professor of Science Education at Miami Universi- 
ty, is framing the measurement questions associat- 
ed with determining whether systemic initiatives 
are making progress toward assuring that schools 
provide access for all students to high-quality 
SMET education. Equity is an important cross- 
cutting theme for measuring progress for educa- 
tional system reform (Zucker et al., 1995). In the 
November 1, 1995, draft of the Systemic Reform 
"Report Card" presented by NSF staff to principal 



investigators of the systemic initiatives, one of six 
drivers of systemic change was given as "Reduc- 
tion in the ’achievement gap’ between the students 
the system serves best and those historically under- 
served." Differential opportunities and performance 
are of concern for a range of categories — ethnicity, 
location, poverty status, gender, and/or the pres- 
ence of disabilities. 

The study of gap metrics is building on the work 
of the Policy Analysis of Systemic Reform project 
and its synthesis of changes in the level and nature 
of student achievement in mathematics and science 
by gender, race, and socioeconomic status. This 
work identifies current metrics used to measure the 
reduction in the achievement gap. Other metrics 
will be identified and analyzed. For example, a de- 
crease in the difference in achievement between 
two groups may seem to represent a move toward 
equity, but in actuality represent a reduction in the 
overall group performance. The difference in the 
percentage of two groups achieving a minimum 
criterion can decrease while the traditionally served 
students accelerate their learning far above the 
minimum. The analysis of the achievement gap is 
complex and requires consideration of multiple 
metrics. 

Expanding the Scholarly Attention Given to Evalu- 
ation of Systemic Initiatives 

A two-day conference of evaluation experts from 
education and other fields will be held in February 
1997. The four thought papers described above will 
be the focus for discussion along with other techni- 
cal issues related to evaluation of systemic initia- 
tives. Proceedings for this conference will be pro- 
duced. 

A University of Wisconsin-based multidisciplinary 
panel was formed in Year 1 to broaden the per- 
spective given to addressing issues of evaluating 
systemic reform. In its four meetings during Year 
1 , the panel members each brought their perspec- 
tives to defining the project’s work and shaping 
relevant issues for evaluation of systemic reform. 
This panel will continue to meet in Year 2, with 
meetings increased in frequency to monthly rather 
than every two months. The panel’s role will shift 
from primarily advisory to more active engagement 
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to bring their members’ expertise as mathemati- 
cians and scientists to bear on project tasks. One 
possible task for the panel is to design review 
procedures for judging the intellectual qualities 
being assessed by evaluation instruments. 

Continuing Efforts 

Alignment of Standards, Frameworks , and Assess- 
ment Systems. A paper, begun in Year 1 and to be 
completed in Year 2, is being written on proce- 
dures for determining the alignment of standards, 
frameworks, and assessment systems. Existing 
alignment procedures used by states will be de- 
scribed, and recommended approaches will be 
extracted. 

Bibliography of References Related to Systemic 
Evaluations. An indexed and annotated bibliogra- 
phy of references related to systemic evaluations 
will continue to be developed and expanded. This 
resource will be available on the NISE Web page. 
Key words pertinent to evaluation, ratings of use- 
fulness, and other means for an evaluator to locate 
relevant information will be identified to provide 
timely access to the growing knowledge base on 
systemic evaluation. 

Special Emphasis Panel on Evaluation. The Spe- 
cial Emphasis Panel on Evaluation exists to pro 



vide technical assistance and recommendations on 
the full range of activities supported under the 
evaluation program of the EHR Directorate of the 
National Science Foundation. The panel was creat- 
ed at the request of EHR as an added activity to 
the NISE. Work has proceeded in a collaborative 
fashion, with Andrew Porter providing the lead for 
NISE and Daryl Chubin and Conrad Katzenmeyer 
providing the lead for EHR. 

Staff 

An active working group has been formed of peo- 
ple from many SMET areas. Norman Webb has 
served as the leader of the project. In Year 1 Dan- 
iel Heck served as the assistant researcher to the 
project. Donald Chambers is assuming this role and 
that of the coordinator of the project beginning in 
Year 2. The panel members and their fields are 
Christopher Anderson (astronomy), Vicki Bier 
(industrial engineering/applied mathematics), Ste- 
ven Bauman (mathematics), Tom Carpenter (math- 
ematics education), Donald Chambers (mathematics 
education/evaluation), Susan Millar (evaluation/an- 
thropology), Senta Raizen (science educa- 
tion/evaluation), Tom Romberg (mathematics edu- 
cation/evaluation/assessment), Pat Rossman (sci- 
ence teacher), John Witte (political sci- 
ence/evaluation), and John Wright (chemistry). 



Professional Development 



Professional development is essential to effective 
educational reform. The EHR goal that every child 
in the United States has access to high-quality 
school education in science and mathematics can- 
not be realized without the availability of effective 
professional development for teachers. The intent 
of this project is to capture the learnings of current 
professional development efforts that will increase 
the knowledge base from both "craft wisdom" and 
disciplined inquiry and make that information 
accessible to practitioners and researchers alike. 

The project has three goals. The first is developing 
a framework for the design of professional learning 
opportunities for K-12 inservice science and mathe- 
matics teachers in order to expand the range of 



alternatives available to professional developers 
beyond the traditional format of workshops and 
institutes. The framework is a decision-making 
process that incorporates a number of central com- 
ponents: a cyclical design process, robust profes- 
sional knowledge bases, professional development 
strategies, context, and critical cross-cutting issues. 
It is a product of the analysis of Year 1 Fellows’ 
current professional development projects. 

The second goal of the project is to create a pro- 
fessional dialogue for elaborating and understand- 
ing the design framework and the issues raised in 
implementing, sustaining, and scaling up profes- 
sional development learning opportunities. 
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The third goal of the project is to create products 
that provide guidance to designers, funders, con- 
sumers, and evaluators of professional develop- 
ment. The first of these is an NISE Brief on Princi- 
ples of Effective Professional Development for 
Mathematics and Science Education, a synthesis of 
the professional development standards included in 
standards produced by various organizations in- 
cluding NCTM, the National Research Council 
(NRC), the National Staff Development Council 
(NSDC), and NCISE. The second is a book to 
describe and illustrate the design framework that is 
intended as a resource for the professional develop- 
ment community in science and mathematics. We 
recognize that, while the book will be a visible 
outcome, more important are the means by which 
professional developers become aware of the cen- 
tral components of the framework and use them in 
designing professional development experiences. 

At the first NISE Annual Forum in March 1996, 
the project team outlined the framework of profes- 
sional development and led case discussions of 
professional development projects. The Forum was 
helpful in refining the description of the framework 
and its implementation, gathering examples and 
illustrations to supplement those currently avail- 
able, identifying people interested in being linked 
to a professional development network in the fu- 
ture, and informing the broader community of the 
work of the project. 

Year 1 Accomplishments 

Our accomplishments in Year 1 included (a) the 
development of a framework for designing profes- 
sional learning opportunities (Goal 1 revised); (b) 
the establishment of a learning community on 
professional development in science and mathemat- 
ics; one important contributor to this was the plan- 
ning of, preparation for, and participation in the 
NISE Annual Forum as the featured project (Goal 
2); and (c) the planning and writing of a book on 
effective professional development of mathematics 
and science education, with the design framework 
as the keystone (Goal 3). 

The framework for the design of professional 
learning opportunities for K-12 inservice science 
and mathematics teachers is a decision-making 



process that incorporates a number of central com- 
ponents. These components are that the design 
occurs cyclically from planning through implemen- 
tation stages, that it is informed by robust profes- 
sional knowledge bases, that it draws on a variety 
of strategies available to professional developers, 
that it occurs within a specific context containing 
various factors that need to be considered, and that 
it addresses several critical issues that cut across all 
professional development. 

The framework is based on an emerging national 
consensus on standards for professional develop- 
ment in mathematics and science education. A 
major part of the professional knowledge base 
informing professional development is an extensive 
body of literature that describes excellent mathe- 
matics and science teaching and its foundations in 
a coherent set of specific beliefs about students' 
knowledge of and ways of learning about science 
and mathematics, about the nature of the disci- 
plines of science and mathematics, and about rele- 
vant teaching methods in science and mathematics. 
Several of the available professional development 
strategies are primarily focused on the mathematics 
and science of the curriculum as teachers learn to 
teach content units, develop and revise new units, 
or immerse themselves in the processes of inquiry 
of science and mathematics or in the world of 
scientists or mathematicians. One of the cross- 
cutting issues concerns the roles of scientists and 
mathematicians in professional development. 

The strategy used to achieve the goal of profes- 
sional dialogue is to establish and develop a learn- 
ing community consisting of people with a variety 
of professional development interests, including 
professional developers, mathematics and science 
educators, scientists and mathematicians, and eval- 
uators. The learning community's purposes are to 
understand the nature of effective professional 
development in mathematics and science through 
reflection on and study of professional develop- 
ment activities, to educate its members about effec- 
tive professional development, to sustain its mem- 
bers as they design and put into practice effective 
professional development, and to be the primary 
dissemination mechanism for the work of the proj- 
ect. In relation to the learning community, people 
differ in the roles they play, the expertise they are 
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able to contribute, the amount of time they can 
spend, and the outcomes they expect to gain. The 
NISE and the project supports a small, active core 
of people, while others in the wider learning com- 
munity have their own means of support. 

Year 2 Activities 

In Year 2 the team is studying (a) the ways in 
* which the design framework and its central compo- 
nents are implemented in different contexts and (b) 
the organizations, structures, and methods that 
serve to facilitate their effective implementation 
and dissemination. 

The project is studying how a small number of 
programs in a variety of different settings use the 
design framework and its central components to 
provide effective professional development. Pro- 
grams being studied are committed to (a) using the 
design framework described in the book, and (b) 
documenting and analyzing their implementation 
efforts. We are paying special attention to the 
similarities and differences that arise between the 
disciplines of mathematics and science. The pro- 
grams studied in Year 2 are different from, and 
chosen to complement, those studied in Year 1. 
Representatives from these programs will work 
with the team in using the central components of 
the book to understand the ongoing cyclical design 
(including implementation) of professional devel- 
opment experiences. There are large projects with 
significant external funding (comparable to the 
work of Year 1 Fellows) and small, underresourced 
programs in order for us to address questions such 
as, What does effective professional development 
look like from the perspective of these projects and 
programs? Does professional development have to 
rely on large-scale, externally funded projects in 
order to be effective? How can a professional 
development program reconcile its local needs with 
national concerns expressed, for example, in vari- 
ous standards documents? An overarching question 
is, Are there professional development activities 
that are so small that they aren’t worth doing? 

The project is facilitating the use of the design 
framework by science and mathematics profession- 
al developers by working closely with professional 
organizations (e.g., NCTM, NSTA, NSDC, Associ- 
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ation for the Education of Teachers in Science 
(AETS) to design learning opportunities at their 
conferences such as extended workshops, and other 
interactive sessions. The purpose of these learning 
opportunities is to raise peoples' awareness of the 
design framework, the role of professional develop- 
ment communities in executing the framework, and 
the range of vehicles used in the implementation of 
effective professional development. The intended 
outcome, beyond the scope of this project, is that 
participating professional developers will use the 
framework in designing their own professional 
development programs for inservice elementary, 
middle, and high school mathematics and science 
teachers. 

The project is fostering the network established in 
Year 1 as a resource for facilitating the implemen- 
tation of effective professional development experi- 
ences. We are studying the role of the participants 
as a primary means of disseminating the central 
components of professional development, as mani- 
fested in the book and case studies. 

The project is studying various means of communi- 
cation among the network of the wider professional 
development community. This mechanism is im- 
portant for disseminating the ideas and products of 
the Professional Development project. Since partic- 
ipants in the community are geographically dis- 
persed, the means of regular communication are 
primarily electronic. Meetings are another means of 
communication: independent meetings, satellite 
meetings in conjunction with meetings of profes- 
sional organizations, and sessions as part of the 
meetings of professional organizations. 

Staff 

The Professional Development Project personnel 
include as team members Susan Loucks-Horsley 
and Peter Hewson (co-Project leaders), Nancy 
Love, and Kathy Stiles; Year 1 Fellows Hubert 
Dyasi, Susan Friel, Judy Mumme, Cary Sneider, 
and Karen Worth; Year 1 Reflectors Josefina Arce, 
Joan Ferrini-Mundy, Deborah Schifter, Vernon 
Sells, Mark St. John, and Iris Weiss; and Year 2 
Fellows Edward Silver, Margaret Smith, and Mary 
Kay Stein. 
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College Level One: Pathways and Effective Practice 



EHR recognizes the importance of high quality 
undergraduate programs in SMET: These programs 
have a substantial impact on EHR goals of provid- 
ing general SMET literacy for all students, prepar- 
ing students for careers in SMET fields and teach- 
ing, and in addressing equity issues. The first year 
of postsecondary education is recognized as a 
curriculum "pressure point" in education. Experi- 
ences in first-year courses greatly influence career 
trajectories and lifelong attitudes toward SMET-re- 
lated fields. The objective of the College Level 
One (CL-1) team is to identify and study critical 
issues related to these courses. 

Year 1 Accomplishments 

The CL-1 team held a workshop in June, 1995, at 
UW-Madison to help organize its efforts. Several 
dozen individuals, representing a variety of stake- 
holder groups, attended the two-day meeting. A 
Workshop Report, "College Level One: Articula- 
tion, Equity, and Literacy Issues," that summarizes 
the conference was the team’s first deliverable. The 
workshop led to interrelated projects within CL-1, 
whose specific activities are described below. Col- 
lectively, we believe that these projects will enable 
our multiple audiences — students, parents, instruc- 
tors, administrators, the NSF, and the public — to 
sharpen their understanding of first-year college 
SMET courses and their impact on SMET educa- 
tion. We are addressing such issues as how these, 
courses can be made more attractive to students, 
add value for those who take them, and enhance 
the likelihood of success across the wide spectrum 
of students who enroll in them. To ensure broad 
participation in and dissemination of our work, we 
have established contacts with many education- and 
SMET discipline-based professional organizations, 
as represented by our Workshop participants. We 
are also collaborating with the NISE Professional 
Development, Communicating with Mass Audien- 
ces, and Interacting with Professional Audiences 
teams. 

Pathways Project 

The intent of this project, led by Walter Secada, 
was to characterize pathways through introductory 



CL-1 SMET courses taken by individuals who 
have entered various SMET-related career areas, 
including, for example, leadership, business, and 
teaching. Toward this goal, we are conducting 
analyses of large-scale databases of individuals as a 
function of their initial careers to see whether 
linear course pathways or nonlinear constellations 
of courses are appropriate metaphors. As part of 
this effort, we are examining articulation issues in 
an attempt to determine the extent to which credits 
and knowledge transfer between courses and across 
institutions. We have initiated campus and national 
efforts to analyze transcript databases that will let 
us explore issues related to pathways through CL-1 
SMET courses. The paths taken by recent graduat- 
ing majors, spanning a range of disciplines, are 
being determined, and these paths will be used to 
help identify a manageable pool of students in 
introductory SMET courses for more detailed 
study. A preliminary study, based on UW-Madison 
student transcripts, examined the impact of intro- 
ductory mathematics and chemistry courses and 
revealed that this methodology has considerable 
potential. On a national scale, we are working with 
Dr. Clifford Adelman of the U.S. Department of 
Education. Adelman has conducted an analysis of 
the engineering path using the High School and 
Beyond college transcript data set. Dr. Lia Brill- 
hart, a CL-1 Fellow, is collecting information on a 
state-by-state basis from two-year institutions on 
articulation procedures and issues. 

Effective Practices Project 

The overall goal of this project, led by Ann Bur- 
gess, was to identify, characterize, and disseminate 
information about a group of effective, sustainable 
practices in introductory CL-1 SMET courses and 
curricula. In collecting information about these 
programs, we are looking for common themes, 
strategies, and philosophies that make the courses 
or curricula particularly effective. Because we are 
selecting a group of effective practices that repre- 
sent the diversity of the CL-1 population, including 
institution type, SMET content area, and student 
populations served, we expect to find a diverse 
group of approaches to reform and measures used 
to define effectiveness. While we intend to remain 
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open to considering new ways to measure effec- 
tiveness, we are relying on traditional measures, 
such as improvement in students’ conceptual un- 
derstanding and problem-solving skills (particularly 
in complex, real-world situations). Other measures 
are students’ attitudes toward math and science, the 
fraction and demographics of students who suc- 
cessfully complete the course or program, and the 
number of students who go on to take a second 
SMET course. 

Through discussions with CL-1 team members and 
others, we have narrowed the focus of our study to 
effective, transportable practices that have been 
sustained. The questions that we are attempting to 
answer are: (1) Will we find common components 
in a diverse group of particularly effective 
first-year SMET programs? If so, what are they? 
and (2) What are the key factors that allow effec- 
tive practices to become institutionalized? Reports 
from agencies that award grants for improving 
college SMET teaching and publications and con- 
ference proceedings describing projects to improve 
introductory SMET courses and programs have 
been collected. 

We have identified programs with which to begin 
our study by interviewing people in each SMET 
field who are "in touch" with national efforts to 
improve first-year SMET programs. We have be- 
gun the process of contacting by phone this initial > 
list of nominees. The reports and phone interviews 
will provide us with more information about the 
specifics of each practice and evidence for its 
effectiveness. We are also exploring key factors 
that led to the program’s initiation and allowed it 
to be sustained. In addition, we ask each practitio- 
ner to recommend other programs we should inves- 
tigate. 

Equity Project 

Our project on equity issues, led by Janice Down- 
er, was driven by considerable evidence that vari- 
ous population groups are differentially impacted 
by SMET courses (National Science Foundation, 
1994). In light of this evidence, individuals teach- 
ing and administering CL-1 SMET courses need 
greater awareness of equity issues and an under- 
standing of strategies for change such that they can 



better facilitate effective interventions for equitable 
participation. To this end, the objectives of the 
CL-1 equity project are to (1) further illuminate 
how certain practices in CL-1 SMET education 
may inequitably impact women, individuals of 
color, and students from low socioeconomic status 
(SES) groups; (2) categorize the current reform 
efforts from an equity perspective; (3) identify the 
underlying organizational structures that are inher- 
ently impediments to equity; (4) identify cultural 
and structural characteristics that have enabled 
successful practices to achieve equity; and (5) 
develop tools that will help administrators, instruc- 
tors, and other stakeholders from a variety of insti- 
tutional settings make informed choices regarding 
reform programs targeting equity. 

Year 1 activities focused on gaining a requisite 
understanding of equity in higher education as it is 
defined in terms of educational opportunities, treat- 
ment, and outcomes and how current practices and 
innovative reform efforts in SMET education may 
have a differential impact on women, individuals 
of color, and students from low SES groups. An 
extensive survey of the literature is underway that 
seeks to identify the reasons that large numbers of 
academically able students avoid or leave CL-1 
SMET courses. The review also will clarify the 
importance of formal support structures, education- 
al practices, and student and institutional attributes 
in promoting student perseverance and success in 
SMET coursework. We are adapting a framework 
developed by Harvey and Klein (1989) for measur- 
ing educational equity as it applies to CL-1 SMET 
courses. Their approach looks at inputs, processes, 
and outcomes, as they involve individual learners 
and groups of learners. 

Year 2 Activities 

Year 1 activities helped establish an internal struc- 
ture and external presence for our activities. In 
Year 2, we are building on this foundation to en- 
hance awareness among the diverse stakeholder 
groups of the key issues associated with CL-1 
SMET courses. Many issues of interest to NSF’s 
Division of Undergraduate Education (DUE), such 
as cross-disciplinary sharing of advances in college 
SMET education, teacher preparation, articulation, 
and equity are embedded in our Year 2 projects. A 
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document providing a longer-term perspective on 
the priorities and objectives of the CL-1 team has 
been prepared for DUE. A key resource for the 
CL-1 team has been a report from the NSF Sub- 
committee for the Review of Undergraduate 
SME&T Education. 

Some reorganization of the CL-1 team has been 
made to help us focus our efforts, most notably the 
merging of the Equity project with the Pathways 
project. The Equity/Pathways project is now under 
the joint direction of Walter Secada and Aaron 
Brower. 

Effective Practices Project 

As baseline information for year two activities, 
CL-1 is conducting, in collaboration with the NSF, 
a literature survey of postsecondary SMET educa- 
tional research. This project will occur in three 
phases. In Phase I, a searchable database of rele- 
vant articles is being constructed and a taxonomy 
developed for its use. The database comprises 
entries from the ERIC database, augmented by 
articles from other leading sources of postsecond- 
ary SMET education research. When completed, 
the database, which will be updated quarterly, will 
be accessible to a wide audience of stakeholders 
via the NISE Web site and possibly by CD-ROM. 
Phase II will occur in two stages. In the short term, 
a few topics of considerable interest, such as prac- 
tices that enhance equity in CL-1 SMET courses, 
will be selected and critically reviewed. A longer 
term component of this phase will be to identify 
other topics, using the database as a guide, and to 
critically review them. Products from these efforts 
will be a series of articles, published through NISE 
and the archival literature, and also accessible via 
the Web site. Phase III will provide a "behind the 
scenes" look at some of the topics by constructing 
a databook in collaboration with the Equity/Path- 
ways Project (see below). 

During Year 2, the Effective Practices project will 
continue to characterize SMET reform efforts by 
focusing on themes and issues that transcend disci- 
plinary boundaries, identified in part through the 
aforementioned postsecondary SMET education 
research literature review project. The products 
resulting from this effort will integrate current 



theoretical perspectives, as found in the education 
literature, with case studies of model SMET reform 
programs. The combination of these perspectives 
will promote discussions between scientists and 
educators as well as provide practical tools to 
guide reform-minded faculty and policymakers. 
Craig Bowen and Leonard Springer, Fellows asso- 
ciated with this project, will assist us in the collec- 
tion and analysis of data on existing reforms. Shei- 
la Tobias, another CL-1 Fellow, will prepare a 
report on the role of CL-1 SMET courses in teach- 
er preparation. 

Equity/Pathways Project 

A databook that Xvill be developed in conjunction 
with the literature review project can reach a vari- 
ety of audiences and call the attention of stakehold- 
er groups to the emergence of a powerful new 
assessment tool: campus databases containing tran- 
script and survey information. These databases are 
now accessible at a number of institutions. 

The databook will contain statistics from a variety 
of disciplines and from two- and four-year geo- 
graphically dispersed institutions that illustrate 
how, for example, changes made in CL-1 SMET 
courses led to quantitative improvement, using 
such metrics as subsequent course performance, 
retention rates, and time-to-degree completion 
statistics. We believe that the document will help 
catalyze the interest of other individuals and insti- 
tutions in making such data available for analysis. 
Moreover, we anticipate it will raise consciousness 
among stakeholder groups about critical CL-1 
SMET issues that include career preparation (in- 
cluding teaching careers), assessment, SMET litera- 
cy, equity, and articulation, as well as describe 
approaches to address these issues where they are 
being implemented. Factors that need to be consid- 
ered in interpreting such data will also be present- 
ed. 

This effort is being followed by more extensive 
transcript analyses, including completion of data 
collection from two-year institutions with the assis- 
tance of Lia Brillhart and initial characterization of 
the national patterns involving transfer into and out 
of these institutions. We are continuing to work 
with Clifford Adelman of the U.S. Department of 
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Education to characterize paths to a variety of 
SMET-related careers from the national transcript 
databases. This research includes use of a new 
database called "Baccalaureate and Beyond," ad- 
ministered through the National Opinion Research 
Center at the University of Chicago; it is designed 
to follow students for a minimum of four years 
after graduation. 

Evidence for differential impact in the context of 
specific courses will be sought through studies 
examining enrollment profiles and performance in 
CL-1 SMET courses for underrepresented groups. 
An additional objective of this activity is to devel- 
op mathematical tools that can be applied to the 
transcript data of a broad spectrum of postsecon- 
dary institutions and used to measure student reten- 
tion through SMET curricula. In parallel with this 



quantitative analysis is a qualitative analysis of the 
interaction of institutions with their student popula- 
tion. We seek to understand how administrators, 
instructors, and students interact with the curricu- 
lum and with each other in shaping objectives, 
allocating resources, and evaluating program effec- 
tiveness of existing support/intervention programs 
within selected institutions. 

Staff 

College Level One Team members presently in- 
clude Arthur Ellis (team leader), Clifford Adelman 
(U.S. Department of Education), Craig Bowen 
(Fellow), Lia Brillhart (Fellow), Aaron Brower, 
Ann Burgess, Sam Donovan, Janice Downer, Abbe 
Herzig, Peter Hewson, Jack Husted, Robert 
Mathieu, Senta Raizen, Walter Secada, Leonard 
Springer, and Sheila Tobias (Fellow). 
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Dissemination Programs 

Interacting with Professional Audiences 



The overarching goal of the Interacting with Pro- 
fessional Audiences team is to ensure that the new 
knowledge generated by the Institute gets into the 
hands of all professionals who can make use of it. 
The team helps Institute researchers identify and 
connect with relevant audiences beyond the aca- 
demic community and develop effective means of 
reaching them. As a result, a much broader range 
of professionals and their organizations will be 
knowledgeable enough to make use of Institute en- 
deavors than is often the case for university-based 
research projects. The three primary goals of the 
Interacting with Professional Audiences team are 
(1) to share knowledge and information generated 
by NISE with key SMET stakeholders, (2) to 
promote interaction and dialogue among SMET 
stakeholders about NISE research directions and 
applications, and (3) to encourage policymakers 
and practitioners to put research knowledge about 
SMET education into action. 

Year 1 Activities 

The first NISE Annual Forum 

The IPA team invested a majority of its Year 1 
effort in making the first NISE Forum a premier 
event in the science, mathematics, and technology 
education communities. The Forum’s purposes are 
to make relevant professional audiences aware of 
NISE work, but, just as importantly, to gain those 
audiences’ perspectives on and participation in 
NISE work. The process of planning and conduct- 
ing the Forum has established an active, collabora- 
tive process within the Institute and with NSF and 
potential dissemination partners for the Institute. 

The first Forum, Professional Development for 
Science and Mathematics Education — Putting 
Knowledge into Action, was very effective. The 
agenda showcased the Professional Development 
team’s project, but also enabled every NISE team 
to introduce its work and solicit input. Because 
several corporations suggested by the National 
Science Teachers Association — Delta Education, 
Duracell, Merck, Toyota — made donations, the 



Institute was able to avoid charging a conference 
fee. A wide range of professionals participated, ap- 
proximately 70% of them from outside Washing- 
ton, even though the federal government shutdowns 
permitted NISE to provide only two months notice 
for the event. The Technical Education Research 
Center (TERC) distributed major NISE Forum 
presentations in audio and text form via the Inter- 
net to a network of professional developers, includ- 
ing all Forum participants. The NISE Formative 
Evaluation team’s interviews and questionnaire 
about the Forum indicated very high participant 
satisfaction with the event. 

Over 1,600 influential individuals were invited to 
the Forum. This large number of people was com 
tacted despite the space limitations of 200 partici- 
pants in order to begin awareness of the Institute. 
The IPA team contacted executive staff and elected 
officers of over 150 organizations, not only those 
in science, mathematics, engineering, technology, 
and education, but also organizations and individu- 
als with a more general interest in and influence on 
science and education policy. The database of 
Forum invitees will be used as a foundation for 
future Annual Forums: it already is being used for 
Institute mailings to announce study results, publi- 
cations, etc. 

Recruiting dissemination partners 

The IPA team staff met with executive directors of 
senior staff of NSTA, NCTM, NRC, and NSDC, 
who agreed to have their organizations become 
Institute dissemination partners, beginning with 
cosponsorship of the first Annual Forum (except 
NRC). These organizations are considering copu- 
blication or secondary distribution of relevant 
reports produced by the NISE and inviting Institute 
projects to make highlighted presentations in their 
annual conferences. For example, the NSTA invit- 
ed the NISE Professional Development team to 
conduct half-day versions of the NISE Forum’s 
agenda during all NSTA regional conventions in 
Fall 1996 and to have a prominent program slot in 
the 1997 annual meeting. 
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Helping NISE teams and management to develop 
dissemination plans 

The IPA team suggested activities to NISE teams 
and management that might be appropriate parts of 
their dissemination plans. Potential strategies in- 
clude innovative activities such as making thought- 
ful use of the Institute’s Web site, gaining cover- 
age by professional newspapers such as Education 
Daily or the Chronicle of Higher Education, and 
working with partner organizations to develop 
articles in their periodicals that tailor Institute 
project results into findings for specific audiences. 
Teams are weighing the advantages of these less 
common strategies against the effectiveness of 
more typical dissemination activities used in acade- 
mia such as presenting at conferences and includ- 
ing articles in refereed journals. As the plans of 
each team for specific products unfold, further 
progress can be made on NISE’s overall five-year 
plan. 

Year 2 Activities 

Organizing the Second NISE Annual Forum 

Organizing the second Annual Form will be the 
largest IPA team activity. The database will be 
revised and augmented to address the agenda focus 
for Year 2. Relevant organizations will be contact- 
ed to cosponsor the event such as the Council of 
Chief State School Officers (CCSSO), the Consor- 
tium for Policy Research in Education (CPRE), and 
the Education Commission of the States (ECS). 

The IPA team will have primary responsibility for 
ensuring that the Forum agenda addresses the 
needs of educators as identified by NSF and other 
key audiences. The IPA team will also manage all 
of the logistics of the Forum, including mailing 
invitations, making personal calls to targeted par- 
ticipants and organizations, arranging for a confer- 
ence site, negotiating meeting logistics, preparing 
conference materials, and making travel arrange- 
ments for presenters and NISE staff. 

Advising and helping NISE teams and management 
with dissemination activities 

The focus in Year 2 shifted from helping the teams 
develop preliminary dissemination plans and 



matching them to the needs of various constituen- 
cies to helping them actually produce the products 
and outputs of their work in forms most useful to 
their various constituencies. The IPA team will 
provide technical assistance in three primary areas: 
(1) giving guidance to teams about the informa- 
tion needs of targeted audiences, e.g., teachers and 
local educators attempting reform; (2) recruiting 
new associations as partners and contacting pub- 
lishers that reach the targeted audiences; and (3) 
helping the researchers to prepare publications and 
presentations for the target audiences. 

For example, once a team identifies work that has 
potential for commercial publication as a book or 
monograph, the IPA team helps by developing a 
prospectus and talking with publishers whose hold- 
ings fit the product and whose marketing bases 
match desired audiences. To illustrate, the IPA 
team advised the Professional Development team in 
preparing a prospectus for its first book. One pub- 
lisher already has expressed very strong interest. 
Since the plans for each team’s specific products 
are still unfolding, the overall NISE Dissemination 
Plan prepared in Year 1 is being managed, re- 
viewed, and revised throughout Year 2 as each 
team makes final determination of its products and 
outreach for its projects. The IPA team is meeting 
with each team leader quarterly to identify the 
dissemination strategies having the highest impact 
from the perspectives of the NISE, NSF, and NISE 
dissemination partners. 

Securing commitments from NISE partners 

The IPA team acts as a liaison between organiza- 
tions and specific teams to share information and 
knowledge from the research that is of particular 
interest to the organizations and to help their con- 
stituencies understand and apply Institute work to 
the reform of science and mathematics education. 
The IPA team also works with the associations to 
get commitments from them to create spinoff pub- 
lications, e.g., articles in journals and periodicals 
that highlight and tailor Institute results to the 
organizations’ constituencies. The goal for Year 2 
is to secure commitments from these partners for 
up to three cosponsored products. 
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In Year 2 we will expand the number of dissemi- 
nation partners to include other science and mathe- 
matics organizations and associations and reach out 
to the policy community to involve them in the 
Institute’s work. For example, we are meeting with 
the scientific societies such as the American Chem- 
ical Society (ACS) and the American Association 
for the Advancement of Science (AAAS), and 
education policy groups such as the CCSSO and 
the ECS. We also are maintaining communications 
and nurturing the working relationships we estab- 
lished with associations in Year 1. 

Design Year 3 study of scaling-up issues 

The IPA team plans to launch a study of the imple- 
mentation issues that surround reform efforts as 
they "go to scale." The field does not know enough 
about what it takes to scale up and how to best 
achieve scale from development efforts. In Year 3, 
the IPA team will initiate a study of educational 
reform models and programs that have effectively 
reached a substantial level of effort. To inform this 
study, in Year 2 the IPA staff will 



convene a panel of dissemination experts, as well 
as program developers and funders. 

Three planning documents are being prepared to 
inform the Year 3 study of Scale-up: (1) a list of 
programs that are believed to have achieved rea- 
sonable scale and some background information on 
how they have achieved this status, (2) a set of 
criteria for analyzing sites believed to have scaled 
up, and (3) a set of study questions that will be 
addressed in the Year 3 research. 



Staff 

The primary staff for the Interacting with Profes- 
sional Audiences tasks are Senta Raizen and Ted 
Britton of the National Center for Improving Sci- 
ence Education in Washington, DC. Other NCISE 
staff assist in conducting specific Interacting with 
Professional Audiences tasks such as organizing 
and recruiting participants for the Annual Forum 
and meeting with association contacts to generate 
interest in the Institute’s work. 



Communicating with Mass Audiences 



The team’s mission has two related components: 

• Developing a better understanding of how to 
use the World Wide Web to promote science 
literacy; 

• Creating greater visibility for the NISE, to 
encourage participation in the Institute and 
engender support for SMET education initia- 
tives through core news support. 

Year 1 Accomplishments 

SMET literacy on the World Wide Web, Phase One 

Our team reached its first major milestone: devel- 
oping and disseminating to the public a Web site 
that promotes SMET literacy by creating informa- 
tion packages pegged to mass media headlines. The 
Why Files (the name is a play on the hit television 
program "The X-Files") combines skillfully written 



text, compelling graphics, timely news photos, and 
strategic linkages to other Web sites to produce a 
powerful communication tool to mass audiences. 

The Why Files (http://whyfiles.news.edu/) is 
uniquely positioned on the Web, because it builds 
on the public curiosity generated by breaking news 
stories. Unlike the preponderance of Web sites that 
either recycle information already published in 
paper or other form or serve as clearinghouses for 
Web sites, The Why Files, created strictly for the 
Web, regularly tantalizes the public with a more 
in-depth look at topics already on their minds, 
thanks to the mass media. A few early examples 
from The Files: 

• When AIDS activist Jeff Getty made national 
headlines by receiving a bone marrow trans- 
plant from a baboon, we knew it was time to 
explore the issue of human-animal transplants. 
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• A highly publicized die-off of monarch butter- 
flies in Mexico prompted us to create an infor- 
mation package that explored animal migration. 

• The 1996 Presidential primaries inspired us to 
develop a package on the statistical underpin- 
nings of political polling. 

• California’s decision to delay implementation 
of requirements for electric cars gave us the 
opportunity to discuss the future of electric 
cars, particularly issues of battery development 
and air emissions. 

• The 1996 Olympics gave as a chance to high- 
light the role of science and technology in the 
games. 

We employ standard Web feedback tools to help us 
evaluate The Why Files. We can track the number 
of site visits and the geographic origins of the 
visitors; the portions of the site that the visitors 
download; and the times of day and dates that the 
site is accessed. We have an e-mail feedback 
mechanism easily used by visitors and have estab- 
lished a Netforum for various lines of discussion. 

To go beyond these basics and conduct a more 
thorough evaluation of the effectiveness of this 
communication product, we have developed a 
formal evaluation plan for The Why Files and are 
embarking on the first stages of that plan. The 
evaluation plan includes setting up test sites (com- 
puter terminals dedicated to displaying The Why 
Files) in schools, museums, and other remote loca- 
tions and using a variety of interview and observa- 
tional techniques to obtain visitors’ impressions of 
the Web site and to learn what Web users attend to 
and why or why not. 

Through this evaluation process, we expect to 
reach our ultimate goal of determining what con- 
stitutes effective Web-based promotion of SMET 
literacy and for whom. We expect to learn whether 
a communications product like The Why Files is 
most likely to be effective in the formal education 
process, or whether it has a more significant role 
as an informal science education vehicle. 



Anecdotal responses to The Why Files confirmed 
our suspicion that Web users appreciate frequent 
site updating. Our staff is committed to producing 
a new lead feature package every two weeks, as 
well as regularly "refreshing" other elements of the 
site (such as Cool Science Images, a collection of 
striking SMET-related photos and graphics plus 
explanatory captions) to keep them interesting. 
Frequent updating supports our goals of tapping 
into themes that are recently in the news and of 
encouraging return visits to the Web site. 

We are beginning to market The Why Files to 
mass audiences, in order to maximize the impact of 
our communications product. Currently, an estimat- 
ed 100,000 people per month visit the site. Exam- 
ples of that activity include: 

1 . Offering the Web listing to individuals and 
organizations that catalogue, review, and publi- 
cize Web sites. Thus far, we have targeted 
more than twenty directories, indices, and 
webcrawlers, sent out three USENET postings, 
and contacted ten national educational organi- 
zations. Many already have indicated they are 
listing us. Microsoft selected The Why Files as 
its Site of the Day, and the NCSA placed The 
Why Files on its listing of new noteworthy 
sites (both around February 24) and visits 
soared. 

2. Approaching authors of other SMET- and 
education-related Web sites to establish hot- 
links. The National Science Teachers Associa- 
tion just included The Why Files as a new, 
interesting site and placed the logo, hot-linked, 
in their site. The Cornell Theory Center Math 
and Science Gateway is another important link 
for us. New links are reported on a daily basis. 

3. Developing press packets for traditional media 
outlets, including color prints of pages and 
Why Files business cards. 

4. Networking with science (and nonscience) 
joumalists/groups and displaying The Why 
Files at conferences. (The Why Files was one 
of two sites highlighted at the 1996 AAAS 
annual meeting session on Web communica- 
tion.) 
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5. Pursuing contacts with informal science educa- 
tors, programs, and institutions. Our first effort 
in this arena was to establish a relationship 
with the Milwaukee Public Museum as a pilot 
effort. We installed a computer terminal at the 
museum to showcase The Why Files and build 
it into their Clue Crew program for children, 
which challenges young visitors to find the an- 
swers to a series of questions in the exhibits. 
The museum is expanding its outreach efforts, 
particularly with respect to the Milwaukee 
minority community, and we will explore other 
museum collaborations to reach museum visi- 
tors of color. 

NISE Visibility: Core News and Information Sup- 
port 

Our team has produced a number of written feature 
articles and media news tips highlighting the cre- 
ation of the NISE. Most of this effort has had only 
in-state impact, although an in-depth feature article 
in the University of Wisconsin-Madison’s alumni 
magazine was circulated nationally and prompted 
some intriguing responses. 

The team created an NISE home page on the Web 
(http://whyfiles.news.wisc.edu) to enhance internal 
communication and increase direct public access to 
the Institute and assisted in the development of an 
NISE brochure. The NISE Web site displays the 
Institute’s strategic plan and organization chart and 
features an NISE roster and selected bios. It offers 
the public information on how to participate in the 
NISE and features examples of media coverage. A 
Netforum was recently added to the site so that 
NISE participants can use cyberspace to communi- 
cate and solicit feedback. 

In an effort to attract some early visibility to the 
NISE and demonstrate a novel informal way to 
promote science literacy, we were inspired by 
College Level One Team Leader Art Ellis to devel- 
op SMET-based brain teasers for university sport- 
ing events. In early Fall 1995, we launched a pilot 
project of SMET-based football-related questions, 
displayed on the Camp Randall Stadium score- 
board at University of Wisconsin-Madison home 
football games. 



The pace of college football games allowed us to 
find appropriate moments to insert the questions, 
known as "Barry’s Badger Brain Teasers" with the 
cooperation of UW Football Coach Barry Alvarez. 
After some time spent perfecting the technical 
delivery of the quiz during the game, the brain 
teasers developed a following with the crowd and 
became a popular feature of the football game 
experience. We publicized the project and earned 
both local and national attention, most notably in 
the December, 15, 1995, issue of Science. 

Buoyed by the success of the football pilot, we 
brought the brain teasers indoors for the men’s and 
women’s basketball season, dubbing the product 
"Dick and Jane’s Badger Brain Teasers" after Bas- 
ketball Coaches Dick Bennett and Jane Albright- 
Dieterle. The new logo, created by our team’s 
graphic artist, featured "Spot," a dog, to accompa- 
ny "Dick and Jane." Technical difficulties (an 
extremely small scoreboard and antiquated comput- 
er support system) made this project more chal- 
lenging to accomplish. The pace of the basketball 
games also made the visible introduction of the 
brain teasers more difficult. By the start of the Big 
Ten season, the brain teasers were largely in place. 

Once we had a full complement of model brain 
teasers, we placed them in The Why Files web site, 
so that others could adapt them at will. Early data 
on visitors show that the "Sports" section is a 
popular place in The Why Files. 

Year 2 Activities 

Using the Web to Promote SMET Literacy 

The World Wide Web is such a new technological 
tool that there is very little literature on the effica- 
cy of its use for mass audience communication, 
and even less on using the technology to communi- 
cate science concepts. However, because of the 
burgeoning development and use of the electronic 
environment of the World Wide Web, it is essen- 
tial to conduct systematic studies of how this new 
medium may be used to foster SMET literacy in 
diverse audiences, including the public at large. 
Moreover, the Web has great potential for study. 
Already, with existing software, one can collect a 
wealth of statistics from visitors who leave click- 
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able trails or road maps of their interests in the 
site. 

By combining human and electronic resources, it is 
feasible to ascertain patterns of Web site use and to 
explore a series of related questions: What propor- 
tion of visitors skim by a site or genuinely engage 
themselves in the process of being informed? Can 
we categorize visitors based on their levels of 
involvement with the site’s information, and how 
might such categorization lead to improvement in 
the design and content of a Web site? What types 
of information, presentation, or other characteristics 
draw visitors to particular parts of a Web site? 

And, of most basic concern to the NISE, is a Web 
site a communication channel that can effectively 
empower people to learn and use SMET-related 
information in their daily lives? 

The Communicating with Mass Audiences team is 
well-positioned to conduct ground-breaking re- 
search on the efficacy of World Wide Web SMET 
communication. The Why Files is developing into 
a site with optimal characteristics for meaningful 
Web research. It features, and will continue to 
feature, many of the latest multimedia presentations 
and takes good advantage of this new communi- 
cation medium; it is regularly updated, a facet that 
encourages repeat visits and offers visitors many 
choices; and it presents timely, news-inspired infor- 
mation that may appeal to a mass audience. 

In Year 2, the team is developing a research effort 
by forging links with respected scholars in science 
communication and drawing on researchers who 
are developing special expertise in the area of 
learning and the new educational technologies. 
UW-Madison Journalism Professor Sharon Dun- 
woody, a national expert on science communica- 
tion is leading this research project. The research 
program has two phases. 

In Phase One, the research team is gathering de- 
scriptive data on Web communication patterns, 
using The Why Files audience members as pro- 
spective study subjects. (We are consulting experts 
on the use of human subjects before proceeding.) 
Research questions will include, Who visits this 
Web site and how often? How do they travel with- 
in the Web site? And what sorts of informa- 



tion/experiences interest or attract them within the 
site? We are seeking out scholars who are already 
looking at these types of issues, both to inform our 
work and to avoid potential duplication. 

In Phase Two, using the initial information gath- 
ered, the team will delve more deeply into ques- 
tions about the effectiveness of communication via 
the Web. In particular, we will explore whether 
and how visitors to The Why Files are empowered 
to understand SMET concepts as a result of their 
experiences with the site. Studies of lay people’s 
attitudes about science repeatedly demonstrate high 
levels of interest but accompanying feelings of 
ignorance. That is, people feel helpless when con- 
fronted with scientific information; they feel that 
they cannot understand it. We hope to see whether 
use of well-designed scientific Web sites amelio- 
rates this sense of helplessness. 

As a result of our inquiry, we hope to be able to 
suggest specific strategies for improving Web 
communication. These strategies will then be tested 
by our Why Files developers and studied again, 
using standard research and development processes. 

Core News and Information Support 

Our primary goal is to increase awareness of the 
NISE by: 

• publicizing newsworthy deliverables; 

• positioning NISE personnel as SMET educa- 
tion experts; 

• aggressively marketing The Why Files to in- 
crease its reach 

Staff 

Susan Trebach is the director of the UW-Mad- 
ison’ s Office of News and Public Affairs (ONPA) 
and the team leader. Terry Devitt, science editor 
for ONPA, is the NISE/Communicating with Mass 
Audiences project coordinator. Amy Toburen, 
associate director of ONPA, works on general 
NISE media relations and is a contributor to The 
Why Files project. David Tenenbaum, a veteran 
science communicator, is our chief science writer. 
Yael Gen, formerly of the Smithsonian Institution, 
is our graphic designer. 
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Organizational Process Programs 

Cognitive Studies of Interdisciplinary Communication 



A major strategy of the NISE is to create interdis- 
ciplinary teams to work on the Institute’s mission 
of improving SMET education. Within the Insti- 
tute, experts from a variety of disciplines form 
working teams of various duration that study sig- 
nificant issues and propose and carry out projects 
related to NISE goals. Such interdisciplinary teams 
are increasingly common in industry, government, 
and society in general and are often utilized by the 
National Science Foundation to guide and facilitate 
its decision-making processes. Many of 
NSF/EHR’s current goals and strategies involve 
promoting interdisciplinary inquiry into the prob- 
lems of education. It is therefore not surprising that 
a substantial body of literature and much "wisdom 
of practice" on interdisciplinarity have emerged in 
recent years (e.g., Chubin, Porter, & Rossini, 1986; 
O’Donnell, DuRussel, & Derry, 1996). Neverthe- 
less, we still possess limited scientific understand- 
ing of how social and cognitive processes interact 
to drive intellectual growth and construction of 
intellectual products in natural interdisciplinary 
groups. Such knowledge might be applied to the 
design of better procedures and technologies for 
improving interdisciplinary inquiry. 

The goal of the Cognitive Studies of Interdisciplin- 
ary Communication team is to understand interdis- 
ciplinary collaboration and improve such collabora- 
tion within the NISE. We are reviewing research 
and conducting observational studies of NISE 
teams that will help us describe, understand, and 
ultimately design and implement new ways to 
facilitate the communicative processes that influ- 
ence productivity and quality of work in interdisci- 
plinary teams. We hypothesize that good collabora- 
tive teams will evolve toward a state of "collective 
intelligence," which entails functioning more like a 
coherent, intelligent organism than like a collection 
of disassociated, independent thinkers. Like other 
researchers (e.g., Smith, 1994), we hypothesize that 
collective intelligence is a type of group behavior 
that might be promoted and enhanced through 
design and use of technologies that facilitate idea 
processing during and between team meetings. 



A unique and important feature of genuine interdis- 
ciplinary collaboration is that it depends heavily on 
face-to-face conversation. Such conversations are 
often lively social events, varying substantially in 
terms of length, intellectual content, and the struc- 
tural and conceptual complexity of arguments that 
are aired. Many of the most substantial intellectual 
accomplishments within collaborative groups — in 
terms of the knowledge that is constructed and the 
questions that are raised — are found in the conver- 
sation that takes place during face-to-face meet- 
ings. Unfortunately, most conversations are ephem- 
eral entities — elegant knowledge structures that 
exist in the air and may largely disappear when 
conferences or meetings end. 

Capturing the knowledge that is constructed during 
intellectual conversation is especially important to 
institutes and conferences of the "think-tank" vari- 
ety, since many experts who contribute substantial- 
ly to such conversations do not instrumentally 
participate in the creation of documents or other 
artifacts that represent the synthesized institutional 
memory of conversational content. Creation of 
such artifacts is often based on methods of note- 
taking that are inadequate with respect to the task 
of capturing the intellectual content of knowledge- 
building conversations. The written artifacts later 
produced from those notes can represent highly 
processed ideas that are substantially edited, selec- 
tively abbreviated, and cognitively biased by the 
individual that produces them. The resulting docu- 
ments may be incomplete and distorted as institu- 
tional memories of meetings and conferences, and, 
as intellectual products, they could possibly be 
inferior to conversations they purport to document. 

For example, in our study of one NISE team, we 
observed that a written summary of what four 
groups accomplished at a working conference was 
carefully designed and constructed by the team 
leadership to serve as an intermediate foundation 
for further knowledge building by that team. This 
was a particularly interesting case of how meetings 
can be thoughtfully engineered by team leadership 
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(using technology and discussion formats) to pro- 
duce intermediate working documents that are 
meant to further the collaborative work of a team. 
We also observed that document preparation was 
labor intensive, that the document could not possi- 
bly be turned around with great speed, and that it 
now exists in a standard written form, which can- 
not be conveniently built on and edited during 
further group conversation. In fact, the document 
has so far had very limited impact on the work of 
the team. We hypothesize, therefore, that a technol- 
ogy might exist or be designed that could greatly 
facilitate team leaders’ efforts to create and build 
on supporting intermediate documents that capture 
intellectual content of interdisciplinary conversa- 
tion. 

The newly developing field called Computer Sup- 
port for Collaborative Work (CSCW) is currently 
attempting to address some of these issues (e.g., 
Smith, 1994). This specialty includes the develop- 
ment of theories about the nature of group intelli- 
gence and the design of technologies that help 
enhance, capture, and preserve such intelligence for 
further use. The work of the Cognitive Studies of 
Interdisciplinary Communication group is related to 
this exciting, growing field. 

Year 1 Accomplishments 

The primary activity of the CSIC team in its first 
year was to initiate intensive studies of three teams 
within the NISE — the Strategies for Evaluating 
Systemic Reform (SESR) team, the Team Leaders 
Team (TLT) and a working research subcommittee 
of the College Level 1 (CL-1) team. We (1) ana- 
lyzed detailed conversational processes and (2) 
studied the genesis and maturation of team knowl- 
edge and functioning over a lengthy period of time. 
Our analyses have primarily focused on collabora- 
tive conversations that can be observed and record- 
ed, tracing their impact on the papers, reports, and 
other products produced by the teams. 

Data analysis began in earnest as we attempted to 
further develop and clarify the theoretical frame- 
work and language we will use for further data 
collection and analysis and to describe our observa- 
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tions. Our approach has been largely qualitative in 
the sense that we are descriptively analyzing team 
processes in detail to contribute to theory about 
effective collaborative work and to further knowl- 
edge about how interdisciplinary perspectives influ- 
ence team functioning. We employed some quanti- 
tative methods in data handling and coding to help 
build a better case for the consistency and validity 
of our observations and interpretations. 

We created three products in Year 1: a literature 
review, a theoretical analysis of the usefulness of 
sociocultural theory for our work, and an in-depth 
analysis of cognitive processes during early devel- 
opment of one NISE team. The literature review 
synthesized research from social psychology, cog- 
nitive psychology, small group research, and other 
fields to highlight the major issues influencing 
cognitive and social processes in interdisciplinary 
teams (O'Donnell, DuRussel, & Derry, 1996). This 
paper is currently available through the NISE. The 
theoretical paper (DuRussel & Derry, 1996) used a 
sociocultural perspective — a theoretical viewpoint 
that implies certain methods and approaches cur- 
rently being debated in the social sciences — as a 
basis for analyzing data from the first few meetings 
of one team. It showed how processes of appren- 
ticeship and use of “common voices” within the 
team contributed to the team's initial development, 
and it also discussed some of the benefits and 
limitations of using a sociocultural approach. A 
short version of this paper was published in the 
proceedings of the 1996 Cognitive Science Society 
Annual Meeting and was presented as a poster at 
that meeting. The third paper (Derry, DuRussel, & 
O'Donnell, in press) explored meetings of the 
SESR team in more depth. It examined how the 
conversation revealed evidence of distributed cog- 
nition, the co-development of ideas among several 
members of a group. It also discussed elements of 
SESR team dynamics and process that prior re- 
search indicates could be relevant to team function- 
ing and effectiveness. This paper was delivered at 
the 1996 annual meeting of the American Educa- 
tional Research Association and will appear in a 
special issue of Educational Psychology Review. 
The topic of the special issue is Distributed Cogni- 
tion. 
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Year 2 Activities 

The following projects will be completed in year 2: 

1 . Analyses of the growth and productivity of the 
Team Leaders Team, the CL-1 working subcom- 
mittee, and the SESR team. 

2. A review and analysis of how technological and 
nontechnological tools for idea processing (such 
as a computer-based white-board system and an 
accompanying conference format for using that 
tool) are now being used to enhance interdisci- 
plinary teamwork in academic and nonacademic 
settings. We will consider whether an appropriate 
tool set (computer system and recommended 
procedures for using it) could be useful to the 
NISE and how it should be designed. 

3. An NISE Brief discussing what the Cognitive 
Studies of Interdisciplinary Communication 
group has learned regarding interdisciplinary 
collaboration and why it is important. 

4. The mini conference described below: 

In November 1996, the Cognitive Studies of 
Interdisciplinary Communication team is hosting 
a small conference on interdisciplinary teamwork 
focusing on understanding collaborative process- 
es and how to manage team-based meetings, 
organizations, and conferences. The conference 
will include faculty participants from the NISE as 
well as invited participants who have special 
expertise or experience pertaining to interdisci- 
plinary collaboration and problem solving and 
who are conducting research on the topic of 
interdisciplinarity. The questions that will guide 
discussions and activities during the conference 
are: 

What is the current theoretical base for under- 
standing interdisciplinary collaboration? 



What is already known about interdisciplinary 
collaboration that can be applied to NISE work 
and beyond? 

What are the best known methods for studying 
interdisciplinary collaboration? 

How can current and developing research find- 
ings contribute to the enhanced performance of 
interdisciplinary teams? 

What technological enhancements and tools 
might be used to facilitate teamwork within the 
NISE? 

How might answers to the above questions help 
other research and educational reform partner- 
ship teams? 

The types of activities to be included in this 
conference are as follows: 

1 . Keynote addresses by Dr. Gavriel Salomon, 
an expert on distributed cognition, and Dr. 
Julie Klein, an expert on interdisciplinarity. 

2. Presentations by other invited researchers and 
authorities on interdisciplinarity. 

3. Small group discussion of key issues in work- 
ing with interdisciplinary teams. 

4. A demonstration of technological tools to 
support collaboration. 

5. A panel discussion on methodologies for the 
study of interdisciplinary teams. 

Staff 

Sharon Derry leads the team. Lori DuRussel, a grad- 
uate student in educational psychology, works as a 
research assistant. Dr. Angela O’Donnell, an associ- 
ate professor of educational psychology from Rutgers 
University and an NISE Fellow, spent six weeks 
during Summer 1995 in residence at the Institute. 

She returned for a brief second visit during February 
1996. Jan O’Neill is a consultant. A part-time re- 
search position is currently being advertised. 
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NISE Formative Evaluation 



The Formative Evaluation team’s primary role is to 
provide internal "formative feedback" information to 
improve the Institute’s functioning as an agile and 
productive organization capable of achieving its 
stated and emerging goals. Secondary roles played by 
the Formative Evaluation team are to produce an 
evolving history of the organization; provide, gather, 
and manage information on organizational activity; 
and engage in collaborative work with various NISE 
teams. 

Year 1 Accomplishments 

Formative Feedback Reports 

Context, Rationale, and Goals. Organizational re- 
search shows that participation in formative evalua- 
tion can significantly enhance the effectiveness with 
which an organization performs. The NISE has cho- 
sen to incorporate formative evaluation processes into 
its culture from the very beginning, with the intent of 
becoming a “learning organization” (Senge, 1990). 

To ensure that these processes are developed and 
maintained at a professional level, the Institute en- 
gaged Dr. Susan Millar, an anthropologist trained in 
evaluation research. Dr. Millar and her Formative 
Evaluation team depend primarily on structured, 
open-ended interviews with all types of individuals 
who hold a stake in the organization. The team also 
collects observational data on most major NISE 
events and a sample of team meetings and collects a 
sample of the written and electronic documents used 
to conduct the daily organizational and research life 
of the Institute. The Formative Evaluation team 
analyzes these multiple sources of information and 
provides feedback in the form of occasional reports 
for the Team Leaders Team and the Management 
Team (see below) and more frequent informal meet- 
ings with the co-directors and individual team lead- 
ers. In all its interactions with Institute participants- 
— during interviews, observations, and “formative 
feedback” sessions — the Formative Evaluation team 
maintains an external (“outsider”) yet proactive 
stance. It seeks to help participants become “reflec- 
tive practitioners” (Schon, 1983) in the art of creating 
a uniquely multidisciplinary and cross-system organi- 
zation designed to improve SMET education nation- 
ally. 



Confidentiality is critical to the Formative Evaluation 
team's success in developing accurate, insightful, and 
consequential analyses of the NISE's organizational 
process. The team assures all persons interviewed 
and observed that the Formative Feedback Reports 
will respect confidentiality and are for use internal to 
the NISE. (By contrast, the other deliverables pro- 
duced by this team are intended for a wide audi- 
ence.) 

Year 1 reports included the following: 

• Formative Feedback Report #1.1: Baseline 
Report on the Team Leaders Team and Man- 
agement Team ( October 1995). The Team Lead- 
ers Team and Management Team used this report 
to help make decisions designed to improve the 
Institute's organizational structure and process 
and to define its emerging sense of organizational 
identity. 

• Formative Feedback Report # 1.2: The First 
NISE Annual Forum ( April 1996). The Forma- 
tive Evaluation team provided two types of eval- 
uation of the First Invitational NISE Forum. 

First, an informal formative feedback report was 
produced for the two teams that developed the 
Forum — the Interacting with Professional Audi- 
ences and Professional Development teams. This 
report evaluated the effectiveness of the early 
Forum materials and agenda, as assessed by 
members of "intermediary" organizations (see 
below). Second, Formative Feedback Report #1 .2 
was provided to the Leadership Team in April 
1996. This report was based on two evaluation 
surveys completed by Forum participants and on 
structured observations and informal interviews 
conducted during the Forum. 

• Report #1.3: Baseline Report on NISE " Inter- 
mediaries ” (April 1996). A third feedback report 
was produced to help the NISE leadership teams 
understand the expectations and needs of organi- 
zations positioned to utilize NISE prod- 
ucts/processes on behalf of students and teachers 
in K-16 institutions. This report was intended to 
help the NISE leadership optimize the NISE's 
ability to pursue projects that are actively valued 
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by these "intermediary" organizations and to 
pursue these projects in ways that enable the 
most effective types of interactions between the 
NISE and these intermediary organizations. 

• Formative Feedback Report #1.4: Report on 
Non-Team Leaders Team/Management Team 
Participants ( April 1996). A formative feedback 
report focusing on the NISE goals and experienc- 
es of Institute participants who are not members 
of the Team Leaders Team or Management 
Team was produced for the Team Leaders Team 
and Management Team in April 1996. This 
feedback report was intended to help the NISE 
leadership teams understand the organization 
from the standpoint of its diverse participants and 
thus enable them to make optimal decisions. 

Year 2 Activities 

Four formative feedback reports are planned for Year 
2. The first of these reports will draw on interviews 
with the members of the Team Leaders Team and 
Management Team conducted in July and August 
1996 and, more generally, on all the Formative Eval- 
uation team’s Year 1 data. This report is intended to 
help the Institute leadership optimize its strategies for 
implementing its Year 2 plans. A second report will 
assess the effectiveness of the Second NISE Annual 
Forum. The third Year 2 feedback report will be 
based on interviews with a sample of NISE interme- 
diaries. At this time, it is expected that this study will 
be designed to help assess how typical members of 
the intended audiences of specific NISE teams are 
responding to the products produced by these teams. 
A final Year 2 feedback report will be based on 
interviews with all non-TLT and MT members of the 
NISE, observations of team meetings and workshops, 
and email and other intra- and cross-team correspon- 
dence. It will be made available to the leadership 
teams in early May 1997 in order to support plan- 
ning for Year 3. 

Story of the NISE 

To achieve and maintain a national scope, the NISE 
requires a research staff that balances a core group of 



long-term participants with a cadre of more short- 
term Fellows and UW-Madison/NCISE participants. 
The "story" document will benefit the core partici- 
pants by providing both a record and a shared under- 
standing of the events that have shaped their organi- 
zation. It will benefit the short-term participants by 
rapidly bringing them up-to-date on events and issues 
that the core participants either take for granted or do 
not remember. It is expected that this document will 
help foster a sense of "NISE identity" in both core 
and short-term participants. Finally, it may be of 
value to NSF staff and others who would benefit 
from an understanding of the organizational history 
and dynamics of the NISE. 

Request for Information (RFI) 

Every organization must maintain accurate and up-to- 
date information about its participants and their activ- 
ities and deliverables. While the task of collecting 
and managing this information is tedious, both the 
organization and its funders require this information 
to assess organizational productivity. The Formative 
Evaluation team has designed a Request for Informa- 
tion (RFI) system that not only obtains and manages 
this information in an accurate and timely manner, 
but does so in a way that makes minimal demands 
on the NISE participants’ time. The reports produced 
from the RFI information submissions will assist 
NISE team leaders, co-directors, and NSF program 
officers working with the NISE in their efforts to 
evaluate and disseminate the processes fostered and 
the products developed by the NISE. 

Staff 

The members of this team include Dr. Susan Millar, 
team leader and Director of UW-Madison’s LEAD 
Center; Dr. Dianne Bowcock, a former school teach- 
er with years of evaluation experience; and Sara A. 
Pfotteicher, who has expertise in evaluation and the 
history of science. 
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Summary 



In the first year of the NISE, we have: 

• initiated eight productive lines of research; 

• recruited a diverse group of Fellows to collabo- 
rate with us in pursuit of our mission; 

• made great strides in creating high quality inter- 
disciplinary teams of researchers who work to- 
gether effectively; 

• established increasingly productive collaborative 
relationships with NSF/EHR’s programs; 

• implemented and modified, where appropriate, 
our proposed management structure, including 
the creation of a highly visible National Advisory 
Board that will meet in Fall 1996; and 

• initiated collaborative work with a wide variety 
of professional organizations in SMET education. 

Our emphasis on dissemination is paying off with, 
for example, a fully subscribed first-year NISE An- 
nual Forum and The Why Files (explaining the sci- 
ence behind the news) on the Web being accessed by 
literally thousands daily. 

Our continuing work is grounded in the following 
ways. First, we are pursuing work that reflects priori- 
ties for the EHR Directorate and is consistent with 
the mission of the NISE. Recognizing EHR’s com- 
mitment to systemic reform, we are pursuing a num- 
ber of efforts that collectively will shed light on 
progress in mathematics and science systemic reform 
and yield insights into how the reform efforts and 
their effects on classroom practice and student 
achievement can be strengthened. Increasingly, EHR 
is confronted with the challenges of satisfying the 
evaluation data needs of external audiences and 
EHR’s formative efforts in program improvement. 
These needs are addressed in our Strategies for Eval- 
uating Systemic Reform work and in convening and 
providing staff support for an EHR Special Emphasis 
Panel on Evaluation. EHR’s major commitment to 



strengthening professional development in mathemat- 
ics and science education is reflected in our work to 
further develop and test a framework for designing 
professional learning opportunities. EHR’s commit- 
ment to reform in undergraduate education is ad- 
dressed through our College Level One work that 
identifies and describes effective sustainable practices 
in introductory SMET courses and that seeks to 
clarify students’ pathways through the SMET curric- 
ulum, especially in regard to equity and subsequent 
careers. To address EHR’s concern for the gap be- 
tween research and development knowledge, on the 
one hand, and education practice, on the other, we 
are pursuing work to close this gap, through both 
interacting with professional audiences and commu- 
nicating directly with mass audiences. To inform and 
strengthen our own commitment to conducting this 
work through interdisciplinary teams, we conduct 
cognitive studies of interdisciplinary communication. 
Finally, we seek continuous improvements in our 
own efforts through our commitment to formative 
evaluation of all NISE projects and products. 

The EHR’s establishment of the NISE has created an 
enormous excitement across the SMET community. 
We have received hundreds of statements of interest, 
requests for information, and visitors to the Institute. 
Our mission is ambitious, but our commitment is 
great. At the end of five years, we expect to have 
launched a whole new approach for the continuous 
improvement of SMET education. The goal of high 
levels of SMET literacy for all segments of our 
population will have become better understood and 
more broadly accepted. New communities of scholar- 
ship and practice will have been established, where 
scientists, education researchers, and education practi- 
tioners work collaboratively to attack the enduring 
problems of SMET education, problems that have 
resisted solutions from more narrow approaches. 
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